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ABSTRACT 

The outcomes and implementation of career academy programs in 
California were examined in a study of one school district's efforts to 
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approximately 10,000 students who were sophomores in the district’s 
comprehensive high schools between 1990 and 1993 were followed through high 
school and several years thereafter. Approximately 14% of the cohort members 
were enrolled in career academies. The data on student outcomes were subjected 
to a multivariate analysis. The academy students were more likely to be female 
and African American. Relatively well developed career academies increased 
their students’ academic knowledge and skills, increased the probability that 
students would attend postsecondary education, decreased the need for later 
remediation in English, and increased^ the probability of university graduation 
for students who were otherwise not likely to even attend a university. The 
career academies were somewhat less successful in increasing students’ 
workplace skills. The following characteristics of successful career academies 
were identified: a relatively complete curriculum; a school-within-a-school 
that sheltered students from hostile or indifferent school environments; large 
amounts of start-up resources; a committed program head; and school and 
community support. (A description of the research and data analysis methods is 
appended. The bibliography lists 15 references.) (MN) 
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Executive Summary 



The basic underpinnings of the educational 
reform movement known as school-to-career, 
or school-to-work, are that students will be 
more motivated to study, will perform better 
in high school, and will fare better in the labor 
market and in higher education if their high- 
school experience is directly connected to the 
world of occupations, careers, industries, and 
professions. 

California has put considerable support into a 
school-to-career model known as the career 
academy since Assembly Bill 3104 was en- 
acted in 1985, which offered competitive 
planning grants of $25,000 to 10 schools to 
initiate these programs, and $50,000 per year 
for initial implementation. Three years later, 
Senate Bill 605 created the Partnership 
Academies program, offering planning grants 
of $15, 000 and implementation grants of 
$67,500 per year. That program is still in 
operation as an extensive network of training 
and technical assistance. In 1999, 180 schools 
in 35 counties received support. 

This approach is the basis for one of the most 
expansive restructuring efforts currently under 
way in California secondary education, al- 
though elements of it have been in place for 
more than a decade. After nine years of 
research and observation of school-to-career 
(STC) programs in one California school 
district, starting in 1991, we sought to answer 
a number of questions related to the ways 
career academies can reform urban high 
schools to increase education and workplace 
knowledge and skills. As part of the process 



we examined high-school transcripts, admin- 
istered post-high-school surveys to several 
thousand students, and undertook site visits to 
academies, schools, and district offices. 

The Career Academy Model 

This educational reform is being driven by 
the transformation of the nation s economy to 
one based on services and information pro- 
cessing, which rewards investments in “human 
capital” more than physical labor. Economic 
and labor-market changes, coupled with the 
perceived failure of high schools to prepare a 
large segment of students for stable employ- 
ment, have set the stage for major changes. 

In theory, by showing students the possibilities 
of employment in high-wage, interesting jobs, 
those who were previously alienated from 
school will see the relationship between their 
studies and a successful life. By integrating 
academic courses with first-hand experience 
in industries that are hiring locally, students 
should leave high school with skills leading to 
employment in the area. Both the increased 
motivation to learn and the acquisition of 
work-placed knowledge and skills should 
increase students’ inclination to stay in school 
and to learn the course material — set in 
accordance with high state and national aca- 
demic standards. 

The career academy is the most thoroughly 
developed STC program model in practice. It 
builds a “school-within-a-school,” integrates 
academic and vocational learning into a spe- 
cific career or industrial focus, and creates a 
cohort of students who take four core classes 
throughout their high school and stay with a 
small group of teachers.This clustering of 
students and teachers creates a strong emo- 
tional, social, and academic support system 



and isolates and “protects” students from what 
may well be an alienating, ineffective school. 

Although career academy models vary across 
the state, they share several core elements: the 
school- within-a-school structure, integrated 
academic and vocational curriculum, and 
employer and workplace involvement. 

This model was instituted in our case study 
district beginning in 1985, the first year such 
programs appeared in California. Its evolution 
from a few idiosyncratic demonstration 
projects to a districtwide strategy framed our 
analysis and helped answer some key questions 
about career academies. In this summary we 
briefly discuss answers to the questions below. 
Other questions are addressed in our complete 
report. 

^ Is this an appropriate educational strategy for 
all students in all schools? 

^ Is it an appropriate educational strategy for 
developing a local labor force? 

^ What components are necessary for a suc- 
cessful career academy? 

^ What is the appropriate scale for career 
academies? 

^ What support is necessary for them to be 
effective? 

The Case Study District 

The school district we studied is fairly typical 
of California’s large, inner-city public school 
systems. It serves over 50,000 students and 
contains six comprehensive high schools. Over 
90% of the district s students are from groups 
considered to be ethnic minorities, more than 
one -quarter have limited English proficiency, 
nearly 40% receive free lunches, and the aver- 
age daily attendance is only slighdy over 80%. 
Educational outcomes for the district s stu- 
dents are generally poor. 



The district s career academy program grew 
out of small, home-grown innovations started 
by teachers in the mid-1980s. The first acad- 
emy was established in 1985, and was in the 
first group to receive state funding. By 1996, 
about 14% of the districts high-school stu- 
dents were in career academies, although the 
proportion of students enrolled in any school 
varied from 4.5 to 50.4%. The largest jump in 
the number of academies came in 1997, fol- 
lowing the adoption of a districtwide STC 
plan. Because the career academy model was 
at the heart of this plan, 20 new academies 
were added in the next two years, raising the 
total to 33 by 1998. 

The district s academies during our study 
period represent the full range of develop- 
ment, from recent startups to several nationally 
known, highly regarded prototypes. They were 
implemented under varying conditions, with 
different levels of completeness, resources, 
dedicated staffing, and school-level support. 
This diversity in capacity among academies in 
the same district is a common phenomenon 
in cities that have embraced the model. 

Although the district established policies to 
ensure inclusiveness and heterogeneity, acad- 
emy students differed from nonacademy stu- 
dents in their demographics, parental 
background, and prior knowledge and skills. 
They were more likely to be female and 
African American and less likely to be Latino 
and Asian than nonacademy students. A 
smaller percentage of academy students were 
from the special education and limited-Eng- 
lish-proficiency populations. These differences 
existed in part because the academies were 
developed in areas where populations had the 
lowest socioeconomic status. 

Thus, enrollment varied greatly among the 
schools and academies, and academies 
occupied somewhat different niches in each 
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schools overall environment. The smallest 
high school, which also had the highest 
proportion of poor students, had the largest 
proportion of students in academies. 

Student Outcomes 

School success. Relatively well-developed 
career academies increased their students’ 
academic knowledge and skills, compared to 
those taking other high-school programs. 
When academy program components that can 
build knowledge and skills are in place, there 
is strong reason to expect the program to raise 
the educational outcomes of many students. 

Career academies in the district increased 
high-school completion rates and made note- 
worthy improvements in several aspects of 
academic achievement beyond high school. 
They increased the probability that students 
would attend postsecondary education, de- 
creased the need for later remediation in 
English, and increased the probability of 
university graduation for students who were 
otherwise not likely to attend, much less 
graduate. 

The increase shown in grade-point averages 
may be particularly important in that it crosses 
a critical threshold for males, African Ameri- 
cans, and Latinos: the difference is between a 
D/D+ average and a C average. Even though 
the average is still low and is hardly an indica- 
tor of postsecondary success, raising the grade- 
point average (GPA) into the C range could 
open postsecondary doors for students. Almost 
all of the academies increased GPA, with the 
notable exception of the two less compre- • 
hensive programs. 

Work success. The district’s career academies 
were somewhat less successful in increasing 
workplace skills. The percent of individuals 
working in a job related to their high-school 



program increased only for females, Latinos, 
Asians, and those who speak English profi- 
ciently. None of the groups had high rates of 
entering fields related to their high-school 
course of study. Even with the positive career 
academy influence, no more than 8% of 
former students work in a field related the 
focus of their high-school program. 

However, relatively few academy students in 
our study were exposed to key work-based- 
learning components of the program. Only 
about 37% had internships, and for those 
students the experience was extremely posi- 
tive. Over 90% reported that the internship 
made them realize the importance of doing 
well in school. The internships appeared to be 
a source of both academic and work-based 
training, with well over 80% of the interns 
using reading, math, and writing skills on the 
job. 

Policy Implications 

Is the career academy an appropriate educa- 
tional strategy for all students in all schools? 
Not surprisingly, academy students got more 
out of the program when they and their 
school put more into it. Successful academies 
raised their students’ academic knowledge and 
skills and also provided a positive social 
climate that independently raised the proba- 
bility of increased learning. These successful 
academies had strong community support, 
long-established funding sources, a well- 
developed curriculum that students actually 
completed, and students who were capable of 
taking full advantage of it. 

Perhaps because the programs studied were 
concentrated in economically impoverished 
areas, academy students who were not gener- 
ally successful by state standards were success- 
ful in comparison with nonacademy students 
in their own schools. Thus, academies that 
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seem to be ineffective according to bench- 
marked standards may be quite effective when 
comparing the educational outcomes of 
students in similar circumstances. Conversely, 
academies that meet local or statewide bench- 
marks may not always have outcomes that 
exceed other courses of study at the same 
school. 

Career academies cannot erase all of the aca- 
demic problems associated with our educa- 
tional and social systems, nor can they erase all 
of the knowledge and skill deficiencies that 
students bring to high school. Clearly, other 
reforms must be implemented in conjunction 
with career academies if all students are to 
leave school prepared to face the labor market 
and educational challenges of the 21st century. 

Is the career academy an appropriate educa- 
tional strategy for developing a local labor 
force? Career academies can decrease the gap 
between the low level of skills that labor- 
market entrants generally have and the rela- 
tively high level of skills employers require if 
the mismatch is defined in terms of academic 
skills. This would occur because academy 
students left high school with greater levels of 
academic knowledge and skills than did 
nonacademy students, and because a greater 
percentage of career academy students 
attended postsecondary education. However, 
academies such as the ones in our study may 
have only a limited capacity to meet narrowly 
defined job-specific skills that local employers 
are seeking (e.g., dental technology). 

Few differences existed between the wages 
and employment of career academy and 
nonacademy students in the first few years 
after they left high school. However, this com- 
munity’s program was less direcdy geared than 
some others toward building a pipeline to 
employment because of circumstances in the 



local labor market. Many other school-to- 
career programs are similar in this respect. 

What components are necessary for a success- 
ful career academy? The most important chal- 
lenge for program planners and administrators 
is to ensure that all the program components 
are set in place. That may sound almost tauto- 
logical, but the experience of school reform 
indicates the need for this warning. Successful 
career academies had several characteristics in 
common, specifically: 

^ A relatively complete curriculum 

^ A school- within-a-school that sheltered 
students from hostile or indifferent school 
environments and provided a social support 
system of teachers and peers 

^ Large amounts of startup resources in terms 
of time (staff and curriculum development), 
equipment, and internships 

^ A program head whose commitment, en- 
ergy, and devotion led and inspired teachers 
as well as students 

^ School and community support, including 
business support for work-based-learning 
components. 

What is the appropriate scale for a career 
academy? The complexities of successful acad- 
emy operation are noted in our complete 
report, and caution against wholesale adoption 
of the model. Among the factors to consider 
in determining scale is the academy’s 
smallness, so students can feel special and 
supported. A school, particularly a smaller one, 
that consists entirely of career academies, may 
face fairly intractable scheduling problems, 
particularly with elective classes. Moreover, 
not all teachers and students are suited to this 
style of teaching and learning. 
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A Final Word 



What support is necessary from partners for 
effective career academies? Strong support for 
the strategy is required at the site prior to an 
academy’s implementation, and throughout its 
lifetime. 

Widespread and genuine support of teachers is 
essential to developing and executing the 
curriculum necessary for the career academy. 
Without support from parents and the commu- 
nity career academy programs will remain 
undeveloped or underdeveloped, because 
opposition or ambivalence will retard their 
growth. Principals and district administrators hold 
the key to scheduling teachers and classes, 
providing funding and guiding student enroll- 
ment, as well as setting content standards, 
curriculum practices, and educational goals. 



Successful career academies represent an am- 
bitious agenda for basic changes in the way 
schools are run. They may be worth the effort, 
as there is evidence that in the right circum- 
stances the new curriculum, social support, 
and work experiences the academies offer can 
make a very important positive difference in 
the lives of many students. The challenge for 
educators is to create and replicate those en- 
abling circumstances for the right teachers and 
students, without losing academic rigor or 
growing so large that the academy’s special 
qualities would be lost. 
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Career Academy Programs in California: 
Outcomes and Implementation 



Nan L. Maxwell and Victor Rubin 



Introduction 

Students will be more motivated to study, will 
perform better in high school, and will fare 
better in the labor market and in higher edu- 
cation if their high-school experience is 
directly and realistically connected to the 
world of occupations, career paths, industries, 
and professions. This is the basic concept 
underlying the reform movement known as 
school-to-career (STC), or school-to-work. 
The premise is at once straightforward and at 
the same time thoroughly controversial. 

Although it provides the basis for one of the 
most expansive restructuring efforts currendy 
under way in California secondary education, 
elements of the approach have been in place 
for more than a decade in selected high 
schools and school districts. Early experiences, 
pilot projects that were sustained and gradu- 
ally replicated in other schools, provide a 
unique window on implementing this new 
way of teaching and learning, and on assessing 
their outcomes for students. 

This report summarizes nine years of research 
and observation of school-to-career programs 
in one California school district. We ask and 
answer the general question, “In what ways 
can career academies reform urban high 
schools to increase education and workplace 
knowledge and skills?” Using high-school 
transcripts and post-high-school surveys of 



several thousand former students in one dis- 
trict and the observations of their teachers and 
principals, we provide California policymakers 
and educators with information about where 
the STC approach has been and where it may 
be headed. 

The driving theme for educational reform in 
STC centers on transforming the American 
economy to one that is based on services and 
information processing, and that rewards in- 
vestments in “human capital” more than 
physical labor. When the changes in the 
economy and labor market are coupled with 
the perceived failure of high schools to ade- 
quately prepare a large segment of students for 
stable employment, the stage is set for major 
changes. 

In theory, by showing them that employment 
in high-wage, interesting jobs is possible, stu- 
dents who were previously alienated from 
school will see the relationship between their 
studies and a successful life. 1 

When the academic courses are integrated 
with first-hand experience of careers in indus- 
tries that are hiring locally, students should 
leave high school with skills leading to em- 
ployment. Both the increased motivation to 
learn and the acquisition of work-placed 
knowledge and skills should increase students’ 
inclination to stay in school and to learn the 
course material. 
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The academy’s clustering of students and 
teachers creates a strong support system 

that isolateskand “protects* f students from \\ 

■ ■ . •>< *$-. •«*>£='* ' 

what may well be an alienating | 

ineffective school 



► What components are necessary to build a 
successful career academy? 

► What is the appropriate scale for a career 
academy program? 

► What support is necessary from partners for 
effective career academies? 



School-to-career reforms also dictate that 
course materials meet high state and national 
academic standards, rather than a norm or 
expectation of low performance, as was often 
the case with traditional high-school voca- 
tional education. 

The career academy is the STC program 
model that is, arguably, the most thoroughly 
developed in practice. The academy builds a 
“school-within-a-school,” integrates academic 
and vocational learning into a specific career 
or industrial focus, and creates a cohort of 
students who take four core classes through- 
out their high school and stay with a small 
group of teachers. This clustering of students 
and teachers creates a strong emotional, social, 
and academic support system and isolates and 
“protects” students from what may well be an 
alienating, ineffective school. 

This is the academy model that was instituted 
in our case study district beginning in 1985, 
the first year such programs appeared in Cali- 
fornia, and it is this period of evolution, from 
a few idiosyncratic demonstration projects to a 
districtwide strategy, that framed our analysis 
and helped answer the following six questions: 

► Is the career academy an appropriate educa- 
tional strategy for all students in all schools? 

► Is the career academy an appropriate edu- 
cational strategy for developing a local labor 
force? 

► If career academies become commonly 
available, what gaps remain in curriculum 
offerings? 



Answers to the first three questions provide 
information to policymakers and educators 
about which students are well served by career 
academies, and answers to the last three pro- 
vide information to program designers and 
administrators about building successful 
programs. 

The Evolution of Policies and Programs 

Responses to the “Skills Crisis” 

The crisis in knowledge and skills was 
brought to national prominence with the 
1983 National Commission on Excellence in 
Education s influential report known as A 
Nation at Risk . The commission characterized 
these deficiencies as fundamental nationwide 
problems of education and economic com- 
petitiveness. The report was blunt and emi- 
nently quotable in its pronouncement about 
the inability of our educational system to 
provide necessary technical knowledge and 
skills or to reverse “a rising tide of medioc- 
rity” and called for curriculum revision such 
that all high-school graduates would master 
the “New Basics” 

The STC reforms initiated during the early 
1990s built upon and broadened the New 
Basics. Educators and employers initiated part- 
nerships to integrate high academic achieve- 
ment with active learning in real-world jobs. 2 

Three pieces of federal legislation codified 
many of the consensus ideas about best prac- 
tices in the new integration of school and 
work. These were “Goals 2000: Educate 



America Act,” the “Carl D. Perkins Vocational 
and Applied Technology Education Act of 
1990” (Perkins II), and the “School-to-Work 
Opportunities Act” of 1994. 

A complete STC system was to provide stu- 
dents with the skills necessary to succeed aca- 
demically in high school and continue their 
education, yet also impart the industrially and 
occupationally specific human capital neces- 
sary for them to succeed in the labor market. 
The belief was that providing work-based 
skills only, in lieu of academic knowledge and 
skills, would reestablish the two-tiered, in- 
equitable education system that had been 
embodied by curriculum tracking. Instead, 
school-to-career reforms would provide both 
work-based and academic knowledge and 
skills to all students. 

Within California, in 1992 the Department of 
Educations High SchoolTask Force issued a 
landmark report entitled Second to None: A 
Vision of the New California High School , which 
identified many of the ideals and strategies 
later embodied in the federal Goals 2000 and 
School-to-Work legislation. The report 
stressed that high schools must be restructured 
to adapt “to the needs of an urban, industrial 
society . . ”The report stated that we need to 

engage our students in strengthened cur- 
riculum; prepare more students for college 
and also to prepare students to qualify for 
) technical preparation programs and jobs; 
develop outcome-based accountability; 
provide effective support and reduce the 
dropout rate; establish an environment of 
professionalism for school faculty; initiate 
effective parent, business, and community 
involvement; and make instructional and 
organizational changes to allow students 
to reach these levels. 

The 1994 federal School-to-Work Oppor- 
tunities Act was largely consistent with this 
vision and initiated a massive planning process 



Within California , a 1992 report 
stressed that high school must be 
restructured to adapt “to the needs of 
an urban, industrial society.” 

in California. The resulting state School-to- 
Career Plan reflects, but moves beyond, the 
experience that the state had already gained 
through its Partnership Academy program and 
its implementation of Perkins II. 

A fully realized school-to-career system would 
require comprehensive changes in how high 
schools teach and classify students. Successful 
curriculum integration would allow high- 
school students to gain workplace knowledge 
and skills without reducing their attention to 
academic achievement. 

The complete STC system would encourage 
active participation by business, postsecondary 
education, and community sectors. The high- 
school programs would be fully articulated 
with community colleges and universities. All 
parties would have access to current labor- 
market information about the skills in 
demand, and the curriculum would convey a 
portable set of skills that had been validated by 
national boards. 3 

Adherence to the standards would provide 
students with the flexibility to pursue either 
employment opportunities or continued edu- 
cation upon high-school graduation. 

The Career Academy Model 

The career academy model has received 
considerable support in California for many 
years, predating the policy initiatives described 
above. In 1985, Assembly Bill 3104 established 
the career academy model by offering $25,000 
competitive planning grants to 10 schools. 
Each site in the initial period received $50,000 
each year to implement the academy. This bill 




was followed three years later by Senate Bill 
605, which created the Partnership Academies 
program: planning grants of $15,000 and im- 
plementation grants of $67,500 per year. That 
program is still in operation, with 180 schools 
in 35 counties receiving support in 1999. An 
extensive network of training and technical 
assistance is also now in operation. 4 

School - Within- a- School Structure 

Academies create structures within schools for 
grades 9 to 12 or 10 to 12 by clustering small 
groups of teachers and cohorts of students 
who stay together for many of their courses 
throughout their high-school career. Typically, 
a team of three to five teachers works with 
approximately 50 students (two classes) per 
grade. 

The school-within-a-school approach pro- 
vides a more personalized and supportive 
learning environment and creates a strong 
peer and teacher support system for working 
hard in school. The clustering of teachers 
enables collaborative work that can focus on 
the educational and developmental needs of 
the students. It also promotes active involve- 
ment of parents in the educational process. 

Integrated Academic and Vocational Curriculum 

Career academies integrate curriculum around 
a career theme in a field for which employ- 
ment is growing in the local labor market. The 
curriculum combines both technical and aca- 
demic content, usually through one technical 
and three academic classes each semester. 
(Technical in this sense means a laboratory or 
a setting in which it is possible to work with 



equipment and procedures characteristic of 
the industry or profession.) 

Career academies expose students to a whole 
range of occupations in an industry and give 
students the option of pursuing advanced 
technical and academic training at the com- 
munity college and university level. Although 
this openness to university education is not 
universal in models across the country, it was a 
central outcome of California s statewide 
planning process. 

Employer and Workplace Involvement ' 

Employer representatives from the academy’s 
career field help plan and guide the program 
and are involved as speakers, fieldtrip hosts, 
job supervisors, and mentors for individual 
students. Students often engage in career 
awareness in the initial year (e.g., attend job 
fairs, take field trips, hear speakers), and con- 
nect with mentors in the second year. Stu- 
dents usually have an internship related to 
their career interests in the summer between 
the junior and senior year, and (at times) un- 
dertake part-time employment during the 
senior year. 

Research Framework and Methods 

With the proliferation and diversification of 
STC programs comes an increased need for 
scrutiny. Preliminary research studies have 
suggested that career academies reduce 
dropout rates, increase postsecondary atten- 
dance, and improve job performance and 
work attendance. 6 

But systematic research is relatively sparse, and 
many key questions have not been addressed. 
If school- to-career strategies are widely im- 
plemented, they will displace existing pro- 
grams and require considerable resources and 
significant retraining of thousands of teachers. 
The stakes are high, and the best time to ask 



tough questions about the potential of STC 
strategies is now, when they are relatively new. 

Our approach to program assessment was to 
examine the growth, implementation, and 
outcomes from career academies in one large, 
urban school district in California. While any 
case study approach to program evaluation is 
subject to the criticism that findings result 
from the idiosyncrasies of one specific district, 
the in-depth perspective on program oper- 
ations, model implementation, and student 
outcomes from this approach cannot be rep- 
licated with large-scale random program 
evaluations. Moreover, our results are highly 
consistent with those of the most recent ex- 
perimental model-based evaluation. 7 

Our research results are based on rich quanti- 
tative and qualitative data. Quantitative data 
include information from transcripts of three 
classes in the district’s high school and of stu- 
dents from the district who applied to or at- 
tended the nearby state university, and survey 
responses to questionnaires administered to 
the students in our high-school population. 

Qualitative data include information from site 
visits conducted over nine years (1991-1999) 
and, in 1999, systematic interviewing of all 
directors of academies (that were at least one 
year old), five of the six high-school princi- 
pals, district program administrators, and vari- 
ous academy partners. We combine the two 
types of analyses — the multivariate analysis of 
student outcomes and the qualitative analysis 
of program processes — to answer our six re- 
search questions. 

Multivariate Analysis of 
Student Outcomes 

We assessed the academies’ potential to build 
postsecondary success for students using indi- 
vidual-level student data to compare academy 
students to others who were not in the pro- 
gram. The analysis focuses on the high school 



Preliminary research studies have 
suggested that career academies reduce 
dropout rates, increase postsecondary 
attendance, and improve job performance 
and work attendance. 



and postsecondary experiences of three 
cohorts of public-school students who were 
sophomores in the years from 1990—1991 
through 1992—1993. 

Fourteen percent of the district’s students in 
these cohorts were enrolled in a career 
academy. We followed these 10,102 students 
through high school with individual-level data 
from the district on grades, demographics, and 
related factors, through their initial foray into 
either postsecondary education or the labor 
market, with a survey sent to them about one- 
to-three years after they left high school 
(about 1,200 former students responded). 

We also followed the educational progress of 
former students from the district in their ef- 
forts to attain a degree from the local Califor- 
nia State University by analyzing the 
individual-level student records of all academy 
and nonacademy students from the district 
who applied, enrolled, or attended that uni- 
versity (about 1,500 students). 

We used these data to assess the career acad- 
emy’s influence on student outcomes in high 
school, in the labor market after the student 
leaves high school, and in postsecondary 
education. Because students enter high school 
with nine or ten years of formal education 
and about 14 years of out-of-school experi- 
ences, all of which affect their subsequent 
academic performance, our analysis statistically 
controlled for initial differences using multi- 
variate methods (primarily ordinary least- 
squares and probit analysis). We also isolated 
the independent influence of the experience 
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Academies in the Case Study City 



of being in an academy from the effect that 
the academy has on raising students’ knowl- 
edge and skills. (See the appendix for details of 
our quantitative methods.) 

Qualitative Analysis of Program 
Implementation 

Our need to understand how the programs 
evolved in the case study community was 
motivated by the central importance of imple- 
mentation issues in education reform. The 
academies both require and instigate the re- 
structuring of many features of a school, and 
they require additional financial and human 
resources. 

If the strategy cannot survive the development 
and implementadon phase in a large school 
district, theoretical elegance or anecdotal per- 
suasiveness may not matter. In fact, sometimes 
the largest barriers to change are not ideologi- 
cal disagreements, but more mundane facts of 
school life such as bell schedules and seniority 
clauses in union contracts. 

Academies that operated in our district during 
the study period represent the full range of 
development, from recent start-ups to several 
that were nationally known, highly regarded 
prototypes. They were implemented under 
varying conditions, with different levels of 
completeness, resources, dedicated staffing, and 
school-level support. This diversity in capacity 
among the academies in the same district is a 
common phenomenon in cities that have 
embraced the model and allowed us to high- 
light the programs strengths and challenges. 




Sometimes the largest barriers to change 
are not ideological disagreements , but 
more mundane facts of school life, iuch 
as bell schedules. 



The school district we studied is fairly typical 
of large, inner-city, public school systems 
throughout the state. It serves over 50,000 
students and contains six comprehensive high 
schools. Over 90% of the district’s students are 
ethnic “minorities,” over one-quarter have 
limited English proficiency, nearly 40% receive 
free lunches, and the average daily attendance 
(the percent of days that a student came to 
school) is only slightly over 80%. Educational 
outcomes for the district s students are gener- 
ally poor by most comparative, aggregate 
standards. 

The career academy program within the 
district grew out of small, home-grown in- 
novations that were started by a number of 
teachers in the mid-1980s. The first academy 
was established in 1985, and was in the first 
cohort of those receiving state funding. Within 
four years, it was joined by academies in five 
other fields. The program continued to expand 
in the early 1990s. One academy was initiated 
in 1991 and two more followed in 1992. 

By 1996, about 14% of all the public high- 
school students were in career academies, 
although the proportion of the student body 
in any school that was enrolled in an academy 
varied from 4.5 to 50.4%. The largest jump in 
the number of academies came in 1997, the 
first full school year following the adoption of 
a districtwide STC plan. Because the career 
academy model was at the heart of the dis- 
trict’s plan for STC in high schools, 20 new 
academies were added in the next two years, 
raising the total to 33 by 1998. 

To support the academy’s growth, the city 
council, sitting as the city’s redevelopment 
agency, appropriated an average of roughly 
$1 million annually for their expansion and 
enhancement, from 1990—1991 through 
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1998-1999. Both the city and district rea- 
soned that, in the short run, increasing the 
academic and workplace skills of the city’s 
students in line with the needs of the labor 
market would lead to students being more 
readily hired and retained by local employers. 
Over a longer period of time, the city’s eco- 
nomic base would expand as firms were 
attracted to the city’s ready supply of ap- 
propriately educated workers. 

This appropriation of funds enabled academies 
to support additional planning and curriculum 
time for teachers, obtain new equipment, 
provide hundreds of stipends for student work 
internships, hire greater numbers of tutors, 
recruit mentors, take more field trips, develop 
more robust advisory committees, and engage 
“industry liaisons” to assist with all of this 
additional recruiting and coordinating. 

The city’s financial support was contingent 
upon meeting performance benchmarks, 
spelled out in each year’s contract. Not sur- 
prisingly, the development of academies often 
centered around achieving the benchmarks 
that were given the most attention — those 
concerned with educational attainment. 

This emphasis on grades, attendance, gradu- 
ation, and similar outcomes stemmed from 
the shared belief by educators, employers, and 
some city policymakers that the local labor 
market was moving rapidly toward a knowl- 
edge-based economy in which fewer higher- 
quality jobs would be open immediately to 
high-school graduates. The result was that the 
career academy programs, especially the older, 
more established ones, became more heavily 
focused on preparing students for higher 
education than on immediate job placements. 

This education-focused vision led to a career 
academy model in which programs were ex- 
pected to increase students’ performance in 
five ways: 



An education-focused vision led to a 
career academy model in which programs 
were expected to increase students’ 
performance in five ways. 

► The rigorous academic standards set by the 
programs, the integrated curriculum, and the 
work-based learning component of the 
program were expected to increase the aca- 
demic and workplace knowledge and skills 
that the students take from high school. 

► The contextual learning environment 
within the academy was expected to in- 
crease the students’ motivation to continue 
to higher education by showing them the 
necessity of education in today’s labor 
market, and by increasing the connection 
between academic content and the work 
itself. 

► The work-based learning component of the 
program and the emphasis on building 
workplace skills were expected to increase 
employment opportunities and wages for 
students as they enter the labor market. 

► The emphasis on the changing requirements 
of the labor market and the need to main- 
tain and increase knowledge and skills in 
demand by employers was expected to en- 
hance students’ interest in and capacity for 
lifelong learning. 

^ The increased knowledge and skills gained 
in high school were expected to increase 
students’ ability to continue into post- 
secondary education and to turn increased 
education into better jobs in the long term. 

Students and Schools 

The district established policies so that the 
career academies would reflect a heteroge- 
neous group of students representing all levels 
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of prior academic achievement. The district 
wanted to avoid enrolling only high achievers 
or, conversely, taking only those with aca- 
demic problems. In theory, all academies had 
the same administrative processes including 
formal application, entrance requirements, 
recruiting procedures, requirements to remain 
in the program, and opportunities, such as 
internships. The districts policies explicitly 
referred to inclusion of students at risk of 
school failure. 

Despite this commitment to inclusiveness and 
heterogeneity, academy students differed from 
nonacademy students in their demographics, 
parental background, and knowledge and skills 
brought to high school. Academy students 
were more likely to be female and African 
American and less likely to be Latino and 
Asian than were nonacademy students. A 
smaller percentage of academy students were 
from the special education and limited- 
English-proficiency populations. 

These differences result in part from the 
focused development of career academies at 
high schools whose student populations rep- 
resented the lowest socioeconomic status. This 
emphasis also meant that enrollment varied 
greatly among the schools and academies and 
that academies occupied somewhat different 
niches in each school’s overall environment. 
The smallest high school, which was also the 
school with the highest proportion of poor 
students, had the largest proportion of its 
students in academies. 



For students who had internships the 
experience was very positive, helping 
them realize the importance of doing 
well in school and providing both 
academic and work-based training 



Once academies were free to set their own 
size, their targets and actual enrollments varied 
widely, in keeping with school size, demand, 
and the teachers’ capacities and preferences. 
The actual enrollments ranged from 85 to 221 
in 1994—1995 and averaged 121 that year. 

The Academy in Practice 

Notwithstanding the importance of the career 
academy model to the district, a complete 
academy program had not yet been achieved 
at all the sites by 1999. Academies did gener- 
ally succeed at keeping their students together, 
with nearly 80% of students in designated 
academy courses enrolled in that academy. 

This may not at first seem noteworthy, but 
getting to this percentage represented a major 
accomplishment, since “school-within-a- 
school” strategies present schoolwide problems 
in the assignment of teachers and students. 

The academy programs were less thoroughly 
implemented in many other respects. For 
example, across the district, academy students 
took an average of only about 13 academy 
semester-length courses, whereas the full 
prescribed academy curriculum constituted 
22 courses over three years. Only 23% of the 
academy students took as many as 20 courses. 
Such incomplete academy course completion 
often resulted from students not being en- 
rolled in the program for the full period of 
time. Only about 28% were enrolled in their 
sophomore, junior, and senior years, the in- 
tended duration of the program. Fewer than 
60% of academy students were enrolled in 
both their junior and senior years. 

Relatively few academy students in our study 
were exposed to key work-based-learning 
components of the program. Only about 37% 
had internships. For those students who did 
have internships, the experience was ex- 
tremely positive. Over 90% of the interns 
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reported that the internship made them real- 
ize the importance of doing well in school. 
Internships also provided both academic and 
work-based training, with well over 80% of 
the interns using reading, math, and writing 
skills on the job. Also, only about 31% of stu- 
dents had a mentor, despite this being the goal 
in most years, and many of the mentors met 
their students only a few times. 

Despite a model that emphasized teacher 
interaction and group cohesion, the teaching 
staffs tended to fluctuate from year to year, 
with only 23.4% of the teachers remaining in 
an academy for all four consecutive years dur- 
ing this study. In every case, an academy had a 
solid core of two or three teachers, some of 
whom had been with the program for over a 
decade by 1999, and who undertook most of 
the extra work. They were typically comple- 
mented by a larger number of teachers who 
rotated in and out for one- or two-year stays. 

The components of the career academy model 
were implemented with varying degrees of 
completeness in different academies. Most of 
these differences can be traced to one of two 
sources: 1) different levels of outside funding 
to each academy, and 2) different school envi- 
ronments with different populations. 

Academy development was not uniform 
among the six high schools. Generally, the 
schools in lower-income areas had the largest 
number of academies, while the two high 
schools in higher-income areas had fewer 
programs, less conformity with the overall 
model, and fewer outside resources. 

It was in the high schools with students from 
the middle of the socioeconomic strata that 
the strongest, most comprehensive career 
academies emerged. Within this stratum of 
schools, the right mix existed of pressure for 
reform and teachers with creative ideas and 



The components of the career academy 
model were implemented with varying 
degrees of completeness in different 
academies, and academy development 
was not uniform among the six high 
schools. 



leadership skills. While overall conditions in 
these schools were still challenging, they were 
not as difficult as they were at the two schools 
whose students came from the lowest socioe- 
conomic status (e.g., dropout rates were not so 
extreme, absenteeism was not so high, and the 
facility was generally more manageable). 

The lowest- income schools faced extreme 
challenges that resulted in uphill battles for 
their academies to maintain a cohesive and 
stable staff, coherent curriculum, full class 
schedule, and adequate facilities. Because of 
these limitations, their academies were not as 
strong or complete as they might have been in 
more favorable environments. In cases where 
schools managed to develop important, effec- 
tive program elements, these tended to be in 
extracurricular activities and in personal sup- 
port for students, rather than in the core 
courses. 

The variation in academy development 
among schools is typical of a large district, 
even where, in theory, the same model was to 
be followed for each academy. However, the 
real world of urban schools does not establish 
conditions congenial to the carefully con- 
structed models of either educational reformers 
or researchers. School inequalities and the lack 
of uniform or constant resources and condi- 
tions may make the effort to understand the 
causes of student outcomes more complex, 
but they are best appreciated not as distrac- 
tions but rather as central factors in this story. 
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Perhaps our most conclusive finding is; 
thaPthe career academypfkodel has^tke 
potential to positively affect educational 
outcomes in California . 



Student Outcomes 

Perhaps the most conclusive finding of our 
study is that the career academy model has the 
potential to positively affect educational out- 
comes in California. Relatively well-developed 
career academies increased the academic 
knowledge and skills of their students, 
compared to those taking other high-school 
programs. When the academy program com- 
ponents that can build knowledge and skills 
are in place, there is strong reason to expect 
the program to raise the educational outcomes 
of many students. 

Our study shows that career academies made 
noteworthy improvements in several aspects 
of academic achievement beyond high school. 
They increased the probability that students 
would attend postsecondary education, de- 
creased the need for remediation in English 
at the university, and increased the probability 
of graduating from the university for a group 
of students who were otherwise not likely to 
attend, much less graduate. 

However, our study also uncovered several 
caveats to these accomplishments. First and 
foremost, career academies must build 
academic knowledge and skills for their 
students. In most circumstances, their positive 
influence on students’ postsecondary achieve- 
ments was through increased academic 
knowledge and skills. Students’ exposure to 
careers and heightened motivation, while 
valuable, cannot be considered a substitute for 
building knowledge. 



Second, the career academy’s influence cannot 
fully overcome the disadvantages that students 
from inner-city public high schools face in the 
university. Reducing high-school dropout 
rates and moving students into postsecondary 
education may be necessary conditions for 
improving educational outcomes, but they are 
not sufficient to ensure success after high 
school. 

Third, career academies do not necessary ben- 
efit all students equally. For example, in this 
analysis, African Americans and Latinos showed 
no direct gains in postsecondary outcomes 
that were associated with being in an acad- 
emy. These groups did make significant gains 
through increasing their academic knowledge 
and skills while in the program, but the ad- 
ditional “independent effect” of being in an 
academy was not significant for them. 

Finally, not all academies improve educational 
outcomes for their students. Academies with- 
out critical program components in place did 
not necessarily increase their students’ knowl- 
edge and skills. Without the complete, or near 
complete, implementation of the model, the 
results will be modest at best. 

In the following sections we present a more 
detailed examination of the findings about the 
influence of academy programs on students. 

Academic Performance in High School 

Academies Raised Student Knowledge 
and Skills 

The career academy increased the academic 
knowledge and skills taken from high school 
by participants, as Table 1 shows. The increase 
in grade point averages may be particularly 
important because it crosses a critical thresh- 
old for males, African Americans, and Latinos: 
the difference is between a D/D+ average to 



Table 1 

High-School Outcomes for Academy and Other Students 



Percent Graduated Percent Now in Job 

GPA High School Related to High School 





Average 


Change 


Average 


Change 


Average 


Change 


Total 

Gender 


2.12 


0.176 


84.5 


1.0 


4.9 


1.0 


Male 


1.97 


0.195 


80.5 


1.1 


4.8 


0.7 


Female 


2.25 


0.225 


86.7 


0.8 


5.0 


1.2 


Race/Ethnicity 


African American 


1.84 


0.224 


78.4 


0.0 


5.7 


0.8 


Latino 


1.91 


0.113 


76.5 


4.6 


4.8 


2.8 


Asian 


2.67 


0.264 


93.7 


no est. 


4.9 


0.3 


English 


English 


2.11 


0.217 


84.9 


0.3 


4.6 


1.1 


LEP 


2.13 


0.202 


82.8 


4.3 


6.0 


0.2 



"Average" shows the mean value for all students, and "change" shows how much the number would increase for an academy student, all 
else being equal. For example, the average GPA for all students was 2.12, and the predicted influence of being in an academy was an 
increase of 0.176, which would result in a GPA of 2.396. Numbers in bold are based on statistically significant (p ^ .05) coefficient estimates; 
those in italics are p < .10. A "no est." means that the equation could not be estimated. (All Asians in academies graduated from high 
school.) 



a C average. Even though the average is still 
low and hardly suggests postsecondary success, 
being admitted to (and achievement in) col- 
lege is often far more difficult for students 
with a high-school GPA below a 2.0. Hence, 
raising the GPA into the C range could open 
postsecondary doors for students. Almost all 
of the academies increased GPA, with the 
notable exception of the two less compre- 
hensive programs. 

Although the career academies were successful 
as measured by GPA, they were somewhat less 
successful in increasing workplace skills taken 
from high school. As Table 1 also shows, the 
career academy increased the percent of indi- 
viduals working in a job related to their high- 
school program only for females, Latinos, 
Asians, and those who speak English profi- 
ciently. None of the groups had high rates of 



entering fields related to their high-school 
course of study. Even with the positive career 
academy influence, no more than 8% of for- 
mer students work in a field related to the 
focus of their high-school program. 

Academies Increased High-school Graduation 

Research consistently has supported the find- 
ing that career academies lower dropout rates. 
In simple comparisons of outcomes, we found 
that former career academy students were 
about 2.5 times less likely to drop out of high 
school than those who were not in academies. 

Approximately two years after leaving high 
school, more than 92% of former academy 
students had a high-school diploma or equiv- 
alent, as compared to 82% of students from 
other programs. However, our multivariate 
findings showed that the career academy 



increased graduation rates over nonacademy 
students only by building students’ academic 
knowledge and skills. That is, academy enroll- 
ment increased high-school GPA, which in 
turn increased the probability of graduating 
from high school. 

Better Attitude Totmrds High School 

Apart from grades, we expected that a positive 
high-school experience rooted in the realities 
of the work world would imbue students with 
some lasting positive attitudes and under- 
standings about how to study, and the value 
of education. 

Some of the qualities that we explored 
through our survey several years after gradu- 
ation indicate students’ capacity to learn (e.g., 
whether their high-school program helped 
them to acquire good study habits and to 
meet deadlines). Other items focused on 
whether they had developed a good under- 
standing of the connection between learning 
and earning (e.g., a positive attitude toward 
education, training, and self-motivation). 

Former career academy students were much 
more likely than other students to report that 
their high-school program provided a set of 
skills that would help them to keep engaged 
in what has come to be known as “lifelong 
learning.” This greater capacity for lifelong 
learning was centered in students who had 
attended the few academies that implemented 
the model more thoroughly and, presumably, 
had provided more social and academic 
support. 8 

Former career academy students were 
much more likely than others to report 
that their high-school program would 
help them to keep engaged in {< lifelong 
learning ” 



Explaining the Academy f s Impacts 

Three sets of interrelated factors help to 
explain the impacts of the academies on stu- 
dents’ achievement in high school. The aca- 
demic preparedness that students bring to 
high school, their school environment, and the 
extent of their exposure to the career academy 
program all affected the outcomes and ex- 
plained the varying capacities of career 
academies to increase GPA. 

Our research revealed differences in student 
backgrounds and in the thoroughness of the 
model’s implementation at each site. The 
academy students v who entered high school 
with higher levels of achievement were more 
likely to take a full load of academy courses 
than those students with lower entering 
achievement scores. For example, 37.3% of the 
students with the highest math achievement 
level upon entry took nearly a full academy 
course program by the time they finished high 
school, compared to only 32.8% of the low- 
est-achieving students. 

The differences in these proportions are rel- 
atively modest, but they mask more substantial 
differences among the various schools and 
programs. There was a striking range in the 
level of outcomes for each academy. With re- 
spect to educational outcomes, the key differ- 
ences among the sites were: 

► The career academy raised GPA in seven of 
the nine programs. 

► The career academy increased GPA at the 
time of leaving high school for students with 
different levels of English achievement in 
10th grade, but were less successful at raising 
GPAs for students at low levels of math 
achievement in the 10th grade. 

► The career academy did not increase GPA in 
every high school, perhaps because of the 
differences in school circumstances. In the 
two high schools where the academy did 



not increase GPA, the programs were less 
well-developed than at the other schools. 
They had fewer integrated or specialized 
courses, fewer community partners, often 
few or no paid internships, and fewer than 
10% of students whose academy course 
work approached a full load. 

► More academies improved GPA for students 
with a nearly full academy course load. Only 
two academies raised GPA in high school in 
cases where students took fewer than 12 
courses. In contrast, all the academies in- 
creased the GPA of students who had at least 
20 academy courses. 

These findings are consistent with the recent 
Manpower Demonstration Research Cor- 
poration evaluation of academies nationally, 
which also emphasized the importance of 
having a relatively complete program and 
long-term experience in order for students 
to realize significant gains in academic 
achievement. 9 

The Academy’s Influence After 
High School 

Postsecondary Education More Likely 

Former academy students were more likely 
than their counterparts to attend post- 
secondary education. The magnitude of the 
difference depends on whether the student 
goes to a two- or four-year institution. When 
attendance rates at community colleges and 
four-year institutions are combined, there is 
only about a 10% difference between academy 
and nonacademy students. 

In contrast, when only four-year schools are 
considered, about a 40% difference exists in 
favor of the academy students. This is im- 
pressive, especially in light of the increasing 
demand for an educated workforce. Also, since 
the community colleges’ entrance require- 
ments are much lower than those of four-year 
institutions, the difference is a very striking 



In the two high schools where the 
academy did not increase GPA, the 
programs were less well-developed than 
at the other schools . 



indicator of academies having fostered aca- 
demic success. 

We were also interested in the extent to 
which the results can be attributed to the 
effects of having been in an academy, inde- 
pendent of the students’ scholastic achieve- 
ment. With the exception of two of the 
strongest sites, we found that career academies 
per se did not independently increase the 
probability of attending college. Instead, they 
build academic knowledge that in turn in- 
creased postsecondary education. 

In addition, as Table 2 shows, career academies 
increase postsecondary education only for 
some demographic groups. Although males 
and Asians show increased enrollment in four- 
year universities after having been in career 
academies, being in a career academy did not 
appear to have an independent influence on 
postsecondary enrollment for females, African 
Americans, and Latinos. 

The route to postsecondary success generally 
lies in the knowledge and skills that career 
academies built. These educational gains are 
the key, and should not be overlooked or 
mistaken for other aspects of the program. 
Career academies cannot afford to substitute 
“motivation” and “career exposure” activities 
for the acquisition of hard academic skills. 
They must build academic knowledge and 
skills in order for students to succeed after 
they leave high school. Motivation and career 
awareness may be valuable tools for getting 
students to learn, but do not appear to be 
effectual without being accompanied by 
academic rigor. 



Career academies cannot afford to 
substitute “motivation” and “career- 
exposure” for the acquisition of hard 
academic skills. 



Mixed University Performance 

Given the emphasis placed in the academies 
on gaining access to higher education, the 
logical next question concerns how well these 
students fared once enrolled. When we exam- 
ined students from this district who entered a 
nearby campus of the state university, we 
found distinct differences between academy 
students and others. 10 

A higher proportion of academy students than 
nonacademy students applied to the university, 
but a somewhat lower percentage were ac- 
cepted. As compared with nonacademy stu- 
dents, the career academy students entered the 
university with an equivalent academic 
achievement in high school (as measured by 
GPA), even though on average their socio- 
economic status was lower. 

This suggests that the academies encouraged a 
wider spectrum of students to apply to uni- 
versity and, not surprisingly, some applicants 
whose grades and test scores were lower did 
not get in.Yet overall, the academies helped a 
greater proportion of students from lower 
socioeconomic strata to get started in the 
university. Their increased academic achieve- 
ment in high school increased their rate of 
university acceptance. 

The career academy also reduced the need for 
college remediation courses in English, both 
on its own and by increasing academic 
knowledge and skills brought from high 
school. While the academy’s influence on 
reducing the need for remediation is encour- 
aging, the rates of remediation were still ex- 
tremely high for students from this district. 



Over one half of academy applicants and 40% 
of nonacademy applicants needed both math 
and English remediation before taking univer- 
sity-level course work. 

Even though the career academy lowered the 
marginal probability of needing remediation, 
70% of the university applicants still needed 
some type of remediation before starting 
university-level course work. This lack of 
preparation for college -level course work 
dramatically increases the time and costs of 
completing a baccalaureate degree. The high 
remediation rates should serve as a warning 
that, overall, high-school programs in this dis- 
trict are not successful at preparing for a four- 
year university. 

Short-term Labor-Market Outcomes 

Career academies had very little impact on 
employment and wages in the years immedi- 
ately after students leave high school, as shown 
in Table 2. This is inconsistent with the states 
goals for the program and, to a much lesser 
extent, the local priorities in the case study 
city. STC programs have often been touted as 
a means to provide local labor markets with 
the skilled workforce that businesses demand, 
to provide students with a variety of transfer- 
able skills within a particular career cluster, 
and (as a result) to raise wages and improve 
employment opportunities. 

Our study shows no evidence that currendy 
operating career academy programs produce 
this outcome in the first two or three years 
after high school. Career academy students in 
general fare no better than their nonacademy 
counterparts in wages after they leave high 
school, although academy enrollment has had 
an effect on increased hours worked for 
females and English speakers. 

This could result from the districts building 
academies that did not have as many strong or 
direct links to job placements with specific 
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Table 2 

Postsecondary Outcomes for Academy and Other Students 





Percent Attending 
2- or 4-Year College 


Percent Attending 
4-Year College 


Hours Worked 


Hourly Wage 




Average 


Change 


Average 


Change 


Average 


Change 


Average 


Change 


Total 


74.7 


0.6 


40.4 


2.4 


27.8 


3.27 


7.34 


-.71 


Gender 

Male 


68.4 


1.7 


35.8 


3.8 


29.9 


3.11 


7.21 


.36 


Female 


78.0 


-0.2 


42.8 


3.0 


26.7 


3.40 


7.41 


-.97 


Race/Ethnicity 


African American 


69.5 


- 2.5 


32.5 


0.9 


30.1 


4.32 


6.65 


-.31 


Latino 


58.8 


4.7 


21.9 


3.2 


29.7 


.58 


7.27 


-.31 


Asian 


85.2 


1.3 


53.9 


6.6 


22.4 


2.40 


8.19 


-1.87 


English 


English 


75.8 


-0.1 


43.9 


4.4 


27.7 


3.76 


7.52 


-.97 


LEP 


69.8 


4.6 


25.1 


1.0 


27.9 


.96 


6.49 


.44 



"Average" shows the mean value for all students, and "change" shows how much the number would increase for an academy student, all else 
being equal. For example, the average rate of enrollment in a four-year college for all students was 40.4%, and the predicted influence of being 
in an academy was an increase of 2.4%, which would result in a percentage of 42.6. Numbers in bold are based on statistically significant 
(p ^ .05) coefficient estimates; those in italics are p < .10. 



employers or industries as academies in some 
other cities. However, because youth fre- 
quently make a number of transitions between 
employment opportunities before settling into 
specific work patterns or a “career,” perhaps we 
should not expect to see immediate employ- 
ment gains resulting from this type of program. 

Although the career academies in our study 
did not increase employment and wages for 
their graduates in their first several years, they 
might do so over a longer period of time for 
several reasons. Over the long term, labor- 
market success may be enhanced because of 
three types of educational improvements: 

1 . Academies may help students in the labor 
market by building the knowledge and skills 
they take from high school. In fact, because a 
higher GPA in high school leads to increased 
wages, and because the career academy in- 
creases GPA, students from career academies 
may ultimately be helped in the labor market. 



2. Academies did significantly increase college 
attendance and the articulation of students 
from high school to postsecondary educa- 
tion, which may ultimately increase their 
employment and wages. 

3. Academies may stimulate interest in lifelong 
learning and an ability to engage in con- 
tinued study. If so, employment oppor- 
tunities and wages of career academy 
students might increase throughout their 
working life as they acquire additional 
human capital. 



Although the career academies in our 
study did not increase employment and 
wages for their graduates in the first 
several years, they might do so over a 
longer period of time. 
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Implementing the Academy Model 

In 1996, the school district formally adopted 
the academies model for organizing its high 
schools and began the process of realizing that 
objective. In a separate document, we present 
the voices of academy directors and high- 
school principals three years into that process 
as they described the components that make 
an academy successful and the challenges of 
implementing the model. The five com- 
ponents they mentioned are: an effective 
human resource system; a supportive admin- 
istration; a usable management and infor- 
mation system; ongoing communication; and 
sufficient resources. In this section we use this 
information to summarize the programs 
strengths and its principal challenges. 

Strengths of the Program 

Perhaps the biggest strength described by 
academy directors, principals, and district 
office staff is the sense of community among 
academy teachers and students. Many directors 
and district managers see a fundamental asset 
of their academy as being the network of 
social support that leads to better educational 
outcomes. For example: 

The academy is a different way of dealing 
with things, connections with the students 
with a particular teacher as a “home base.” 
Then they buy into the program, the 
school, themselves. The academy answers 
the question for the students of “Is there a 
safe place for me?” 

The academy is a vehicle for all com- 
munity stakeholders to work together to 



Perhaps the academy ’s biggest strength is 
the sensed of community among academy 
teachers and students. 



improve student achievement in a person- 
alized small learning community, (district 
office staff member) 

Academy directors confirmed this feeling with 
a rating of program components. A “small 
group of faculty working together” was rated 
as highly important by 94.1% of the academy 
directors, and 88.2% regarded as highly im- 
portant a “small group of students taking 
classes together” Nine of the 18 directors 
talked about the collaboration of teachers in a 
small learning community as the academy’s 
major asset, especially the common prepara- 
tion time spent with other teachers in devel- 
oping an integrated curriculum and discussing 
interventions for individual students. 

Academy directors also saw the program’s 
work-based learning component as a major 
strength. Ten of the 18 directors thought one 
of the best things about the academy program 
was providing students an opportunity for 
hands-on, experiential learning through in- 
ternships, job shadowing, or lab classes. As one 
director stated: 

The hands-on experiential learning is 
really an essential part of the program. I 
think it is one of the most exciting things 
the kids can be involved in because that is 
really going to put some meaning in what 
they are learning. 

Perhaps the strongest support statements came 
from the principals, who have fewer vested 
interests in maintaining the programs than the 
academy directors or the district office. Two of 
the five principals thought the academies had 
a positive impact. One commented with cau- 
tious optimism about program expansion: 

So far the academies are working well, 
and if we can get all the teachers to 
understand and know the concept it 
should be quite successful. The structure 



of the academies is a good one; it is just a 
matter of getting the teachers motivated 
to work together as well as the students. 

Challenges 

Of course, many practical problems exist in 
implementing the academy program, par- 
ticularly because of its placement within a 
larger high school and school district. Perhaps 
the most pressing problem is that of schedul- 
ing. In fact, when a key district program staff 
member was asked to identify the three major 
problems facing academies, the answer was 
“scheduling, scheduling and scheduling.” 

Institutional constraints in the labor agree- 
ment, such as six-period scheduling and the 
clause that teachers have only two different 
classes to prepare each semester, place limits 
on the capacity and flexibility of student and 
teacher schedules. Limiting scheduling to only 
six periods presents problems because students 
cannot always schedule the four academy 
courses, and limiting teacher preparations to 
two can mean that teachers must teach in 
more than one academy (e.g., a teacher of 
government and history might have to teach 
in two academies since not enough govern- 
ment and history courses exist within one 
academy). Scheduling common times for 
faculty to meet and prepare material is diffi- 
cult. Scheduling multiple aspects of the cur- 
riculum also poses difficulties: 

Some of the challenges we face are 
scheduling these [school and work-based] 
activities accordingly.The directors and 
teachers work with the School-to-Career 
Liaison to coordinate and schedule the 
activities, but many times it can be so 
overwhelming for everyone involved. 
Transportation issues also pose concerns, 
being that the buses do not run through- 
out the day. However, I do believe that 
these activities enhance the school and 
[that it is] an advantage to the students to 



Perhaps the most pressing problem in 
implementing the academy program is 
that of scheduling 

get a reality-based exposure to the real 
world. 

Administrative support and leadership was 
another key area identified as critical in imple- 
menting an academy. Virtually all the directors 
(94.1%) ranked “district support” as being of 
high importance for the success of their acad- 
emy. An academy director, while praising the 
schools principal, eloquendy stated the posi- 
tive potential of administrative support: 

Now, because of the continuity we have 
been able to enjoy under new leadership, 
the new principal, he’s the only principal 
I’ve worked with that IVe actually learned 
from. We are really a team. If we can build 
a team, we can delegate to other teachers. 
Otherwise, the first few years we were 
reinventing the wheel every year. When 
you have a good principal he can do [a 
lot] for you. When he came along we had 
less staff turnover. We were losing good 
people left and right under the other ad- 
ministration. And they were undermining 
our program. One administrator told me, 

“I don’t know why you are working so 
hard, we might not let you do this, you 
know!’ I stood up and said “you must be 
crazy,” I was so mad. But with this prin- 
cipal we were allowed to do whatever we 
wanted. He wasn’t afraid that is what 
good leadership does. 

Unfortunately, few academy directors had 
such praise for administration. Only 37.5% 
of the directors thought they had a high level 
of district support — the third-lowest score of 
the 17 components. Because it was rated as 
being so critically important, district support 
had the widest discrepancy between high 
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A major challenge for the academies is 
raising resources to support their 
increased cost. Career academies require 
extra resources to succeed. 

importance and low incorporation of any 
item on the survey 

District officials suggested that their perceived 
lack of support by academy directors reflects 
inconsistencies between the two groups’ views 
of their roles and responsibilities. Whatever the 
underlying reason for this discrepancy, our 
research suggests that providing sufficient 
support for academies is a critical challenge. 

Finally, a major challenge for the academies is 
raising resources to support their increased 
cost. As special programs distinct from the 
regular high-school program, career academies 
require extra resources to succeed (such as 
funding for internship stipends or for teachers’ 
extra time to create an integrated curricu- 
lum). However, a key to the academies’ long- 
term growth and health is their gradual 



institutionalization into the school district’s 
core operations, rather than their replication as 
externally supported programs. 

The patterns of expenditures and services 
from 1990 to 1998 can be characterized by 
three broad generalizations: 

1. In the early years, the costs of starting an 
academy and developing its curriculum were 
funded with city support. 

2. During later years, as the district assumed 
these responsibilities, the extra time that 
teachers spent implementing the model was 
not fully compensated. For example, many 
academy directors suggested that managing 
an academy remains a stressful fuH-time-plus 
job, whether paid or not. Many academy 
directors reported working long uncompen- 
sated hours, suggesting a high burnout rate. 

3. Economies of scale are realized with pro- 
gram expansion, although problems are 
encountered if the program “goes to scale.” 

As Figure 1 shows, the additional cost of edu- 
cating a student in a career academy fell from 



Figure 1 

Redevelopment Funds Budgeted for Academies Per Student 




Numbers reflect the total amount of the municipal redevelopment budget for the academies for each year divided by the total 
number of students who were enrolled in participating academies that year. 



Figure 2 

Change in Allocation of Redevelopment Dollars to Academies Over Time 



A: 1991-92 Budget 
(SI, 201 ,000) 



B: 1994-95 Budget 
($1,198,000) 
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based Learning 
20 % 



Equipment & 
Facilities 
22 % 



Adminstration 
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Adminstration 
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Curriculum 
57% 



Internships & 
Other Work- 
based Learning 
44% 




Teaching & 
Curriculum 
45% 



Equipment & 
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3% 



Data are from contracts between the school district and the city for each program year. Teaching & Curriculum includes certificated 
salary and benefits, teacher substitutes, in-service payments, instructional supplies, tutor benefits, and consultants. Equipment & 
Facilities includes noninstructional supplies and equipment. Internships & Other Work-based Learning includes student stipends, 
field trips, transportation, and industry liaisons. Administration includes printing and consultants. 



nearly $3,000 to about $574 as curriculum 
became developed and teachers learned the 
model (Figure 2). However, expansion creates 
a host of other problems, including increased 
scheduling conflicts. 

Internships, an integral part of the career acad- 
emy program, are expensive and the need for 
their funds does not diminish over time (Fig- 
ure 2). In fact, as academy startup costs de- 
crease, the cost of internships increase and 
become a greater portion of the additional 
program costs. In our case study city, intern- 
ships were offered at little or no cost to em- 
ployers, which hampered the development of 
a tradition of employer-paid internships. 

Policy Implications 

Career academies are a potentially effective 
strategy for education reform, and academy 
programs can facilitate postsecondary success 
for inner-city, public high-school students. 
Notwithstanding a positive conclusion at this 



most general level, three important caveats are 
in order: 

1. Career academies are effective only when 
they build students’ academic skills. Program 
designers should resist a quick-fix approach 
that substitutes career exposure or work 
experience for serious academic preparation, 
or that channels students into highly special- 
ized job-specific training without broad 
grounding in basic academic skill s. 

2. Not all career academies are equally effective 
at building students’ academic knowledge 
nor do their programs enhance learning for 
all students equally. Diverse influences are not 
necessarily a problem in a large school dis- 
trict, but administrators must understand that 
(a) every academy needs certain basic 
components and commitments to make a 
real impact, and (b) not all groups of stu- 
dents benefit equally from career academy 
programs. 

3. Career academies, by themselves, cannot 
compensate for inadequate education prior 
to high school. Even where the career 
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academy increased student academic out- 
comes, these were relative measures. Overall, 
the performance indicators were low by 
statewide or national standards. Career acad- 
emies have not yet been shown capable of 
providing all types of students with the 
knowledge and skills needed to succeed in 
college, and their record of overcoming the 
academic achievement deficits that students 
bring to high school is not yet persuasive. 

With this brief summary, we now answer the 
six questions we raised at the beginning of this 
report. 

Implications for Students 

Is the career academy an appropriate educational 
strategy for all students in all schools? Not 
surprisingly, we showed that academy students 
got more out of the program when they and 
their school put more into it. Successful acad- 
emies not only raised the academic knowl- 
edge and skills of students, they also provided 
a positive social climate that independently 
raised the probability of increased learning. 
These successful academies had strong com- 
munity support, long-established funding 
sources, a well-developed curriculum that 
students actually completed, and students who 
were capable of taking full advantage of the 
curriculum. 

Perhaps because the programs in the study 
were concentrated at schools in neighbor- 
hoods with low socioeconomic status, acad- 

Other reforms must be implemented in 
conjunction with career academies if all 
students are to leave school prepared to 
face the labor market and the educational 
challenges of the 2 1st century. 



emy students who generally were not success- 
ful by state standards were successful when 
compared to nonacademy students in their 
own schools. 

Academies that seem to be ineffective when 
held to benchmarked standards may be quite 
effective when their students’ educational 
outcomes are compared to those of nonacad- 
emy students in similar circumstances. Con- 
versely, the student outcomes of academies 
that meet local or statewide benchmarks may 
not always exceed those of students taking 
other courses of study at the same school. 

Academies may be undervalued if they are 
held to statewide standards but operate in very 
dysfunctional schools, or if they have student 
populations that are not prepared for high- 
school academic work. Initial knowledge and 
skill deficiencies cannot be remediated for all 
students in a three-year period, especially if 
the school environment is not conducive to 
learning. This is not to say that high academic 
standards should not be developed and main- 
tained. Indeed, our study findings strongly 
suggest that maintaining high academic 
standards is a necessary condition for post- 
secondary success. 

Career academies cannot erase all of the acad- 
emic problems associated with our educa- 
tional and social systems, nor can they erase all 
of the knowledge and skill deficiencies that 
students bring to high school. Clearly, other 
reforms must be implemented in conjunction 
with career academies if all students are to 
leave school prepared to face the labor market 
and the educational challenges of the 21st 
century. 

Is the career academy an appropriate educational 
strategy for developing a local labor force? Ca- 
reer academies can decrease the skills mis- 
match if the broadly defined mismatch refers 



to the general gap between the low level of 
skills that labor-market entrants generally have 
and the relatively high level of skills employers 
require. Such a convergence would occur 
because academy students leave high school 
with greater levels of academic knowledge 
and skills than nonacademy students, and be- 
cause a greater percentage of career academy 
students attend college. 

In contrast, if the skills mismatch is narrowly 
defined as the mismatch between the job- 
specific skills that local employers are seeking 
(e.g., skills in dental technology) and the skills 
of new entrants, then the career academies 
may have only a limited capacity to alleviate it. 

We found few differences between the wage 
and employment of academy and nonacademy 
students in the first few years after completion 
of high school. Academy students did not 
move directly into positions left unfilled by 
skill mismatches. However, because of the 
local labor market, this community’s program 
was less directly geared than some others to- 
ward building an employment pipeline. 

If career academies become commonly available , 
what gaps remain in curriculum offerings? If 
career academies are not for all students and 
are not effective at all sites, our research find- 
ings challenge administrators to determine the 
students and areas in which the program 
should be offered. Although our data are in- 
sufficient for such a detailed assessment, our 
findings raise several considerations. 

Individual learning styles may determine 
whether or not academies are appropriate for 
all groups of students. A commonly held per- 
ception among school-to-career advocates is 
that 70% of students learn best when the 
learning is in context. This means that as many 
as 30% of the students may learn better in 
more traditional settings. 



We found that not all groups benefitted 
from career academy programs, making 
us cautious about universal adoption. 

For the 70% who learn best in context, the 
labor market may not be the most compelling 
context for all of them. Instead, some students 
may benefit from a different type of integrated 
or contextual situation. Because our study 
found that not all groups benefitted from 
career academy programs, we should be cau- 
tious about universal adoption. 

Some students may need to prepare for jobs 
immediately after high school or are not 
suited for college studies. Traditional curricu- 
lum tracking and the traditional vocational 
curriculum was tailored to their needs. If the 
benefit of career academies lies in their ability 
to increase academic knowledge and skills and 
the probability of attending college, then these 
academies may not provide enough direct 
support for such students. 

Although most school-to-career advocates 
think both options should remain open and 
available through the same program, our case 
study suggests that the challenge of serving 
both objectives well may be hard to resolve — 
particularly in a community where academies 
are strongly oriented toward postsecondary 
education. Moreover, because universal rele- 
vance is often ascribed to the academy model, 
such a challenge may be difficult to acknowl- 
edge direcdy. 

Implications for Program Design 
and Administration 

Site administrators need to consider the fol- 
lowing aspects of the career academies 
program in designing and building successful 
programs.A high level of implementation 
requires sustained, creative, inclusive, and 
systematic planning. 
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What are the components necessary to build a 
successful career academy? 

The most important challenge for program 
planners and administrators is to ensure that 
all program components are set in place. That 
may sound basic, almost tautological, but the 
experience of school reform is rife with im- 
pressive labels covering little of substance. 
Successful career academies had several 
characteristics in common, as noted below. 

► A relatively complete curriculum. Principals 
and program directors can support the staff 
and curriculum development process. With- 
out that support, courses may be labeled as 
being part of an academy, but not be sub- 
stantively changed. Both academic learning 
and motivation suffer as a result. 

► A school-within-a-school that shelters stu- 
dents from hostile or indifferent school envi- 
ronments and gives a social support system 
of teachers and peers. The appreciation of 
and loyalty to the “community” of the more 
effective academies is very significant, and 
clearly contributes to positive outcomes. 
Students, teachers, academy directors, and 
principals consistendy praised the virtues of 
the small learning community provided by 
the career academy 

► Large amounts of startup resources in terms 
of rime (staff and curriculum development) 
and internships. Clearly, several important 
cost factors must be considered when imple- 
menting career academy programs, especially 
if internship wages are included. The extra 
staff time needed for preparation and man- 
agement is significant, as are the equipment 
needs of programs geared to the latest tech- 
nologies. 

► A program head whose commitment, en- 
ergy, and devotion lead and inspire other 
teachers as well as students. The key question 
of institutional development is how suffi- 



cient numbers of skilled, motivated teachers 
can be identified, nurtured, and supported. 
To some extent, having the model incorpo- 
rated into the day-to-day fabric of schools 
reduces a great deal of stress. Nonetheless, 
leadership and creativity are still very neces- 
sary qualities. 

► School and community support. The more 
effective academies had active advisory com- 
mittees whose participants did most of their 
work outside of the meetings, and who 
could be counted on for personal help and 
organizational resources as well as financial 
contributions. In the district we studied, 
parental and community support were in- 
strumental in initiating and developing ca- 
reer academies in schools, and business 
support was instrumental in supporting their 
work-based-learning components. 

When a new approach to teaching and orga- 
nizing high schools moves beyond the experi- 
mental or demonstration-project stage and 
becomes widely adopted, the basic systems, 
incentives, and organizational culture of the 
schools and central district office need to in- 
corporate the characteristics required by the 
model. Integration becomes central to overall 
curriculum development and training, not to 
only a small group of teachers in a high 
school. 

What is the appropriate scale for a career 
academy program? 

How widespread should the adoption of the 
career academy strategy be? The potential has 
prompted a number of districts to convert 
entire high schools to collections of acade- 
mies. Our findings on the complexities of 
successful academy operation should give 
pause to the idea that career academies should 
be widely implemented within a school dis- 
trict or be “wall-to-wall” within a school. The 
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following points should be considered in de- 
termining the scale of the program. 

Much of the success of career academies may 
lie in their smallness. Small programs can 
create an intimate “school- within-a-school” 
environment in which students feel special 
and supported. It is unclear whether this same 
environment can be created if an entire school 
is comprised of career academies. In fact, 
creating wall-to-wall academies would create 
a hierarchy of academies within a school, and 
possibly replicate the kinds of negative dis- 
tinctions associated with curriculum tracking. 

Career academies are expensive undertakings 
when fully realized. Subsidized work intern- 
ships are very costly if provided for the ma- 
jority of academy students. The amount of 
additional teacher and staff time to fully 
realize the model is substantial, especially in 
the first several years. Of course, the long-term 
strategy may be one of cost reductions based 
on incorporating academy functions within 
the core mission and budget of the school 
system so that the costs become central to the 
running of the high schools. 

In some situations, the employers may pay the 
interns from their own budgets. Once the costs 
are borne by any party, the cost-effectiveness 
question is particularly relevant. If the educa- 
tional benefits of the academy are not greater 
than those of traditional programs, even small 
additional costs are not worth the investment. 

A school that contains only career academies 
may face fairly intractable scheduling prob- 
lems, particularly with elective classes, and in 
smaller schools. The solutions include reduc- 
ing the number and diversity of electives avail- 
able to a given student, combining students 
from two academies in the same class, and 
assigning the same teacher to two or even 



Much of the success of career academies 
may lie in their smallness. 

three academies. These arrangements lose 
much of the curricular focus of having the 
academy in the first place. Other conundrums 
of scheduling exist that will take serious work 
to overcome in a program taken to this scale. 

There may not be place for teachers and 
students who do not want to be in academies. 
When part of a schools curriculum lies out- 
side of the academies, teachers and students 
who are not suited to this style of teaching 
and learning can be shifted to nonacademy 
courses. This is not possible if schools or 
districts consist entirely of academy programs. 
This also raises the possibility that significant 
numbers of teachers who do not have the 
interest, focus, and commitment that the pro- 
grams need will be assigned to them anyway. 

What support is necessary from partners for 
effective career academies? 

It is important to ensure support for the ca- 
reer academy approach at the site prior to its 
implementation, and to continually build that 
support over time. Successful programs require 
support from the constituencies below. 

Widespread and genuine teacher support is 
essential to developing and executing the 
career academy curriculum. Such programs 
will not be successful unless teachers believe 
that an integrated, challenging curriculum can 
and should be the means for students to 
achieve postsecondary success. No analyses, 
case studies, or administrative edicts can match 
the influence of teachers who genuinely relate 
their classroom experiences to peers who take 
on something new, unfamiliar, and perhaps 
threatening. 
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Without parental and community 
support , career academy programs will 
remain undeveloped or undeveloped } 
because opposition or ambivalence will 
retard their growth. 

Similarly, parents and others in the commu- 
nity must believe that career academies can 
facilitate students’ postsecondary success. 
Without their support, career academy pro- 
grams will remain undeveloped or under- 
developed, because opposition or ambivalence 
will retard their growth. Career academies that 
were developed in schools believed to be most 
in need of reform, remained much less devel- 
oped than those in schools that had somewhat 
higher socioeconomic status and overall col- 
lege attendance rates. Unless the school com- 
munity is convinced that reform is necessary 
and that career academies are the appropriate 
vehicles for that reform, change will not 
occur. 

Principals and district administrators must 
support career academy development. Site 



administrators hold the key to scheduling 
teachers and classes, providing funding, and 
guiding student enrollment, and will do so in 
the way that fits their vision of instructional 
leadership. District administrators decide the 
focus and emphasis of staff development days 
and set content standards, curriculum prac- 
tices, and educational goals, based pardy on 
past practice and pardy on the current dis- 
trictwide reform strategies. 

A Final Word 

These conditions for the success of career 
academies represent an ambitious agenda for 
basic changes in the way schools are run. 
However, career academies may well be worth 
the effort, since the right circumstances, the 
new curriculum, social support, and work 
experiences offered by the academies can 
make a very important, positive difference in 
the lives of many students. The challenge for 
educators is to create and replicate those cir- 
cumstances for the right teachers and students, 
without losing academic rigor or growing so 
large that the special small-school qualities of 
academies would be lost. 
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APPENDIX 

Methods and Data 



Our evaluation of educational reform includes analysis of both institutions and individuals. We employ 
a qualitative assessment of the programs development and implementation at both the district and 
program level, as well as a quantitative assessment of individual student outcomes. In our qualitative 
assessment, we follow the districts effort to implement a uniform career academy model along with 
the development of nine unique career academies from 1990-1996. In this analysis we uncover many 
necessary ingredients for developing successful career academies, as well as many impediments to their 
growth and implementation. 

Our quantitative assessment of individual student outcomes uses data from the population of high- 
school students who were sophomores in the districts comprehensive high schools between 1990 and 
1993. We followed this cohort of about 10,000 students through high school and through their first 
several years after high school. Because about 14% of these students were enrolled in career academies, 
our study is a rare opportunity to follow a large population of students in career academies and com- 
pare their high-school and postsecondary experiences with those of their contemporaries who did 
not enroll in career academies. 

In general, the qualitative analysis focuses on programs’ development and implementation, while the 
quantitative analysis augments and supports its conclusions.The quantitative analysis focuses on stu- 
dent outcomes, while the qualitative data contextualizes and supports its conclusions. What follows is 
the general framework that is used in each analysis. Keep in mind that, although the discussion here 
presents fairly distinct qualitative and quantitative methods, the analysis undertaken reflects a blending 
of approaches. 

QUALITATIVE ANALYSIS 

We examined the development and implementation of career academies using a qualitative frame- 
work grounded in the reporting that was done for the city government-sponsored evaluation of the 
academies. The reports that were written as local evaluations were circulated locally for several years, 
and the findings were subjected to extensive feedback and recommendations from teachers, admin- 
istrators, community partners, funders, and local policymakers. 

Analytical Framework 

The city government provided targeted funds to the school district for the academies, and the per- 
formance-based contract governing these funds required an annual independent evaluation. Because 
these yearly evaluations overlapped the period during which our population of students was in high 
school, its findings and data can be used to determine the programs evolution over five school years. 
Evaluation occurs at two levels: program and district. 

At the program level, aggregate student outcomes from each career academy (relatively stable) were 
compared annually against city-mandated benchmarks in several areas: academic achievement, pro- 
gram enrollment and retention, attendance, and graduation. We expected academy outcomes to 
improve as an academy matures, develops its programs, and gains resources. Where program-level 
outcomes did not improve or meet city-mandated benchmarks, site visits and interviews with acad- 
emy directors, principals, district personnel, and community partners helped identify problem areas. By 
tracing the evolution of career academies through the eyes of students, teachers, and administrators, 
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and by tracking the changes that occurred in the outcomes, management, and structure as the dis- 
trictwide program expanded, we assessed the program’s viability and utility. 

The yearly evaluation also included indicators of the management of individual programs, such as 
curriculum, master schedules in the school, requisitions, schedules of field trips and speakers, and as- 
signment of mentors. Each year, documentation included the programs’ progress in meeting city and 
district-set objectives in these areas and the factors that may have enhanced or inhibited timely and 
effective implementation. This gives us a sense of the degree of institutionalization that occurred. 

At the district level, the program objectives set by the city and district also specified several important 
steps for developing academies’ curriculum, undertaking long-range planning, and generating com- 
munity support.These objectives included securing of other funding and in-kind resources, and ex- 
panding the academies program throughout the district. Each year the evaluation analyzed the extent 
to which the new programs were implemented according to the model, and noted examples of 
effective implementation as well as possible barriers. In all cases, the approach taken was not to create 
a single “grade” or summary judgment about the new materials and institutional arrangements, but to 
work with the district and the city to develop criteria against which progress could be measured from 
all parties’ perspectives. Three levels of assessment of the district’s progress toward meeting these ob- 
jectives were undertaken: 

1. Timely progress: Were the tasks specified by the objectives completed on schedule or, if not, 
did the district document the impediments to the timely completion of the tasks? 

2. Quality and effectiveness: How well did the new plans and materials meet the needs of the 
students, teachers, and administrators? How much new support was created for the academies 
within each high school, in the district administration, and in the community? 

3. Change: How did the most recent years’ outcomes compare to previous years? 

Qualitative Data Sources 

The complexity of developing, implementing, and administering a career academy — a school-within- 
a-school within a districtwide bureaucracy — is daunting. Many requirements of the career academy 
model are difficult to accommodate within the traditional school framework. To assess whether each 
career academy is institutionalized within the school, or is at least moving in this direction, we evalu- 
ated a large array of data. During our annual evaluations, we conducted dozens of visits to school sites, 
attended training meetings, conferences, and staff meetings, and conducted numerous formal inter- 
views and informal conversations with teachers, administrators, community partners, local policymak- 
ers, funders, providers of technical assistance, and students. Each year we surveyed interns and 
supervisors for feedback on work-based learning, the Work Experience Surveys. We draw on these 
data sources for the qualitative analysis of career academies. 

On-Site Interviews 

To analyze the administrative and management support structures for the career academies, we con- 
ducted on-site interviews of 18 academy directors, five principals, and the district office. (The one 
remaining principal did not return phone calls and faxed requests for an interview.) Each on-site visit 
to the academy directors consisted of a conversational interview that focused on a set of questions 
about structure and current status of the academy program; future of the program; issues in school- 
based learning; and issues in work-based learning. 

Each of the on-site conversational interviews with the five principals focused on a set of questions 
about current operations of the academies in the context of the school; administrative issues for 
academy programs; and issues of program expansion. 




The on-site conversational interview with the districts School-to-Career director focused on the 
same questions that were posed to the academy directors and principals. The director was specifically 
asked to respond to these questions from the district office s perspective and to address the responses 
from the academy directors and principals. (Their aggregated responses were stripped of all remarks 
that could possibly be used for identification and were provided to the director prior to the inter- 
view.) Site guides for the interviews were developed by the evaluators and given to the district for 
review Extensive training was provided to the four site interviewers, who were employees of the 
evaluators’ research institutes. 

As part of this data-collection effort, we asked all the directors to complete a questionnaire asking 
them to rank 17 different components of the academy’s program in terms of their importance and the 
degree to which they were incorporated. The 17 components were listed in the contract between the 
school district and the city and were divided into four categories: 11 

► program goals (build work skills for labor-market entry; build academic skills for college at- 
tendance; build skills necessary for articulation into a community college program); 

► school-based learning (integrated academic and vocational curriculum; articulation agree- 
ments with higher education; contextual teaching within a career focus; team teaching); 

► work-based learning (career exploration; mentoring at the work site; job shadowing, field 
trips, integrated internship; school-based enterprises); and 

► support (district support; small group of faculty working together; small group of students 
taking classes together; business-community support). 

Academy directors were asked to use a five-point scale to measure the importance and incorporation 
of each program element. 

We examined each program component along two dimensions: the percentage of academy directors 
who rated the program component as “highly important” (4 or 5 on a scale of“not essential” to “most 
essential”) and the percentage who related it as “highly incorporated” (4 or 5 on a scale of “not inte- 
grated” to “well integrated”). Thus, we determined for each program element the percent of academy 
directors who believed that it was an important part of the program and the percent who believed it 
was well incorporated into their program. 

Local Evaluation Files 

The local evaluation files contain data in two primary areas: administration and management, and 
academy development. The data are valuable for conveying not only the characteristics of the acade- 
mies and their evolution over time, but also the differences that can arise between a program specified 
on paper and the actual program the students experienced. 

Academy Development. We have fairly extensive information on the availability and use of the 
academies’ curriculum. Annual scheduling information of all classes offered in the district allows for 
yearly comparisons of the availability of and enrollment in academy-designated courses. For example, 
the composition of the teaching teams in each career academy gives us the degree of staff continuity 
or, conversely, the rate of staff turnover. The proportion of enrollment targets attained, in terms of 
students admitted into the program and the enrollment of the students in the designated courses, gives 
us insights into the academy’s growth. 

Additionally, in 1992—1993 and 1993-1994, the curriculum of eight of the nine career academies was 
examined extensively. Teachers were assigned to prepare curriculum guides following the district s 
prescribed format, which required three sections for each curriculum guide: program materials, course 
materials by grade level, and integrated projects. The program materials included sections on the over- 
all philosophy behind each career academy program. 
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Additionally, curriculum guides were to include materials for every academy course at each grade 
level (10th, 11th, 12th). Finally, each career academy was asked to develop projects that integrated the 
academy courses and the career themes. The school district administration expected at least three 
integrated projects for each grade level by the summer of 1994, the point at which the measurements 
of completed work were taken. 

Work Experience Surveys 

As part of the city’s evaluation effort, yearly surveys were administered to the student interns and their 
intern supervisor and, in two of the years, to students’ mentors. The set of surveys of interns had the 
highest response rates. This instrument included basic information about the internship (e.g., the in- 
dustry, occupation, work hours, and relationship to respondents’ career goals); the availability of train- 
ing; an assessment of skills used on the job (reading, math, writing — both use and level); the quality of 
the job as a learning experience (e.g., their interest level, use of abilities, development of skills); the 
relationship of this work to school; and a comparison of this internship to other employment they 
may have had. 

Parallel surveys were administered to each intern’s supervisor, although response rates were lower. That 
survey asked about the supervisor’s relationship to the student, interactions with district personnel, the 
supervisor’s assessment of skills used (reading, math, writing, computers — both use and level), and a 
comparison of the student to other students and employees. 

Finally, for two years (1993 and 1994) a small percentage of mentors assigned to the students were 
surveyed. Mentor surveys included information on the type of interactions (e.g., activities, counseling), 
possible outcomes of mentoring (e.g., personality, educational change in student), and the mentors’ 
satisfaction with program, training, and experiences. While the surveys of mentors further our under- 
standing of the program, the low response rate and the relatively low percentage of students who had 
mentors (about 25%) preclude its analysis. 

QUANTITATIVE ANALYSIS 

Our quantitative analysis examined the career academy’s influence on 1) knowledge and skills ac- 
quired in high school, 2) postsecondary education, and 3) work outcomes at a point about two years 
after high school. We estimated value-added educational production functions that specify knowledge 
and skills taken from high school as a function of career academy enrollment, initial knowledge and 
skills, and controls. In other words, educational outcomes are modeled as a function of inputs into 
the process. 

We distinguish between the youth and adult labor markets when modeling postsecondary outcomes. 
In contrast to adults, young people move frequently within and between work and educational expe- 
riences. This movement means that traditional human-capital models do not always predict youth 
activities. Instead, education and labor-market outcomes during the first two or three years after high 
school often result from high-school experiences and demographics, including family background. 
This is reflected in all of our analyses. 

While the key independent variable in all analyses designates whether a student was in a career acad- 
emy, all analyses include controls for race and gender because education and labor-market outcomes 
vary along these lines. All estimations also control for limited English proficiency and enrollment in 
special education, because of the (presumed) lower levels of productivity that these characteristics 
hold. Perhaps most importandy, all analyses control for knowledge when students entered their high- 
school program. 

In estimations of outcomes from high school, we use the standardized achievement score in the 
sophomore year to measure initial-period knowledge. In estimations of postsecondary outcomes, we 
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use high-school Grade Point Average (GPA) when the student left high school to measure initial- 
period skills. Controlling for initial-period skills is extremely important because it is often correlated 
with subsequent activities (e.g., high -school program, attending postsecondary education) that affect 
outcomes. As a result, their omission as control measures can bias estimates of program impacts. 

All multivariate equations were estimated under three different sets of specifications: 1) using a binary 
career academy variable to indicate a student’s enrollment in a career academy; 2) using a vector of 
binary variables to indicate a student s enrollment in a specific career academy; and 3) stratifying 
equations by school, student’s entering test scores, or level of academy coursework. 

These alternative specifications allow us to assess 1) the aggregate impact of the career academy ap- 
proach; 2) the impact of each specific academy; and 3) the impact of the career academy within the 
same school environment, for students with relatively equivalent levels of preparation, and for ex- 
posure to an academy’s curriculum. 

Knowledge and Skills Acquired in High School 

We first examine whether career academies increase knowledge and skills taken from high school. 
Because achievement tests were not administered to all students in the 12th grade, we use GPA to 
measure knowledge and skills taken from high school. This use of GPA draws two potential criticisms, 
however. 

First, career academy teachers could have different grading standards from those used in the school as 
a whole. In this case, GPA, as a knowledge and skill measure, is invalid if career academies artificially 
increased a career academy students GPA. To examine for this possibility, we compared grades in 
academy courses and grades in equivalent nonacademy courses. With only one exception, there was 
little evidence that the career academies grade on a different standard from the rest of the school. In 
fact, in all but these two academies, analysis suggests that grades in career academy courses are lower 
than grades in comparable courses offered outside of the career academy. 

Second, schools could attach a different value to the same grade. Although relative heterogeneity exists 
among the schools in our sample as compared to samples that have schools from different districts, 
heterogeneity in grading standards could still exist. To control for this potential, we use the individuals 
GPA that has been divided by the school average GPA as our dependent variable. Because this mea- 
sure captures the career academy’s impact (for example) on raising GPA (knowledge and skills) above 
the school’s average, it controls for different grading and achievement that may exist among the schools. 

Our value-added educational production function that links knowledge at the time the student leaves 
high school with career academy program enrollment is therefore specified as: 



1 : 



GPA j - GPA school = a 9 + £a 9 Acad, + £a 9 Demo k + ^K&S, 



where: 



GPA-GPA 



school 



Academy = 



Demo = 
K&S = 



the individual’s grade point average divided by the average grade point average 
at the school, 

1) a vector of binary variables that indicate the career academy in which the re- 
spondent is enrolled, or 

2) a binary variable that indicates enrollment in a career academy, 

a vector of demographic variables that controls for the respondent s gender, 
race/ ethnicity, English proficiency, and special education status, 

a vector of variables that measures the respondent’s academic achievement in 
the 10th grade. 
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Because the dependent variable is constructed as deviation from the school mean, estimation of this 
equation answers the question, “Does the career academy program increase GPA above the average of 
the school?” We use Ordinary Least Squares (OLS) to estimate the equation. Significant and positive 
coefficients on the career academy variables with the inclusion of all control variables suggests that 
career academy enrollment increases the academic (ct g > 0) skills taken from high school. 

In equation 1, career academy students are compared to all nonacademy students in the district. When 
equations are stratified, different comparisons are made. Stratification by school shows how career 
academy students compare with nonacademy students in their school. This controls for many of the 
unobservable environmental factors that could bias estimations. Stratification by entering test scores 
compares career academy students with nonacademy students who have the same level of academic achieve- 
ment when they enter high school. This controls for much of the unobservable individual heterogeneity in 
achievement that could bias estimations. Finally, stratification by academy course work shows how 
career academy students fare with different levels of academy “treatment.” 

Postsecondary Education 

To assess whether the career academy increases postsecondary education, we ask the question, “Does 
the career academy 1) directly increase postsecondary education and interest in lifelong learning, or 2) 
indirectly increase both of these outcomes through academic skills acquired in high school?” To answer 
these questions we estimated educational production functions with the dependent variable measur- 
ing educational outcomes: 

2: Educ = a e + ^a e ] Acad ] + ^a e k Devr\o k + a®GPAj 
where: 

Educ = a vector of 1) binary variables with 1 indicating that the respondent achieved the designated 
level of education, and 

2) categorical variables indicating the respondent s interest in lifelong learning, 
GPA = the individuals GPA when leaving high school. 

The first set of dependent variables are cumulative measures that capture whether a student completed 
or is progressing toward a particular educational milestone: high-school graduation, attending a two- 
or four-year college, and attending a four-year college only. This analysis assesses the ability of the 
career academy (or high-school program) to help individuals advance to the next level of education. 
The second set of dependent variables measures the respondent s perceived preparation for lifelong 
learning that the high-school program afforded. 

We use a series of probit analyses to estimate educational benchmarks and learning measures that are 
constructed as binary variables. Ordinary Least Squares produces inefficient estimates of program im- 
pacts in such cases. We use OLS to assess the impact of career academies on categorical lifelong learn- 
ing measures. If the coefficients on the career academy variables are positive and significant (a c > 0), 
the career academy program increases education directly. If these coefficients are not significantly re- 
lated to education (a c = 0) but do significantly increase academic skills in high school (a g > 0 from 
equation 1), which in turn increase education (ctj > 0), the impact of the career academy is indirect, 
and operates by increasing human capital in high school. 

Should indirect effects be present, we confirm their existence by re-estimating equation 2 without 
GPA as a control. If career academies significantly increase educational outcomes indirectly by in- 
creasing knowledge and skills taken from high school, the coefficient on academies will be significant 
without inclusion of GPA and insignificant with its inclusion. 
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The difference between a direct or indirect effect could be important for the approach taken by, and 
outcomes expected from, school-to-career reforms. If career academies exert an indirect effect on 
postsecondary outcomes, their ability to facilitate postsecondary outcomes lies in their ability to in- 
crease students’ knowledge and skills. The route to postsecondary success is through building academic 
abilities; and the career academy is a mechanism through which these abilities can be strengthened. 
Without increasing knowledge and skills, the career academy will not facilitate postsecondary success 
unless direct effects exist. 

A direct effect, in this case, implies that the career academy facilitates postsecondary outcomes through 
some mechanisms other than enhancing academic knowledge and skills. It could be, for example, that 
career academies increase the students’ motivation or that they build job or education networks that 
facilitate the transition from high school into either work or postsecondary education. In either case, 
such impacts would be independent of acquiring academic skills in their own right. 

Postsecondary Work Experience About Two Years After Leaving High School 

We also examine the career academy’s ability to increase positive labor-market outcomes about two 
years after leaving high school. More specifically, we ask the question “Does the career academy 1) 
directly increase postsecondary labor-market outcomes or 2) indirectly increase outcomes through acad- 
emic skills acquired in high school?” The work outcomes of interest include hours worked, wages, and 
self-perceived workforce preparation. For estimations of hours worked and perceived workforce 
preparation we estimate: 

3: Work = a h + ^a^Acach + a^GPAj + Ja h k Demo k + a^Grad + a^Enr 

where: 

Work = 1) the average numbers of hours that the respondent worked or 

2) a vector of variables that measures perceived workforce preparedness, 

Grad — a binary variable indicating whether the respondent graduated from high school, 

Enr = a binary variable indicating whether the respondent is enrolled in classes. 

For estimations of wages, we include hours worked (Hrs) on the right-hand side to control for the 
impact of differential labor supply and estimate: 

4: Wage = a w + ^a^Acad. + oG'GPAj + X aV k Demo k + aX TG rad i + a^Enr + a^Hrs 

where: 

Wage = the respondent’s (log) hourly rate of pay 

Each of these estimations (3 and 4) assesses the ability of the career academy to increase labor-market 
opportunities during the school-to-career transition period. We use probit analyses to estimate equa- 
tions with binary measures of workforce outcomes, and OLS to assess the impact of career academies 
on hours worked, (log) wages, and categorical workplace-preparation variables. 

If the coefficients on the career academy variables are positive and significant (a! 1 > 0), the career 
academies increase labor-market outcomes direcdy. If these coefficients are not significandy related to 
labor-market outcomes (a h = 0) but do significandy increase academic skills in high school (a. 8 > 0 
from equation 1), which in turn increase work (aj 1 > 0), the career academy’s impact is indirect, and 
operates by increasing human capital in high school. 
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Educational Experience at the University 

Finally, we are interested in observing the career academy’s influence on each successive step through 
the university: entrance into, route through, and exit from. By examining the academy’s influence on 
the progression through each step, we can determine the points at which it helps students through 
college and the points at which it has no influence. It may be at the latter points that other interven- 
tions are needed. We examined the university outcomes and activities of individuals from our district 
who applied to the local university from 1990 to 1998 for a degree program. 

We first assess the academy’s influence on entrance into college as measured by acceptance to and en- 
rollment in the university. This analysis answers the questions, “Does the career academy help students 
gain admission to a four-year university?” and “Does it help ensure that students enroll once admitted?” 

We then assess the academy’s influence on the route taken through college by examining the need for 
remediation and the transfer status of students. Undertaking remediation and transferring to the uni- 
versity, usually from the community college, often lengthen the time for students to earn a degree. 

This analysis answers the question, “Does the career academy help the student move through the uni- 
versity in a timely manner?” 

Finally, we examine the academy’s influence on the student’s exit from the university as measured by 
dropping out, academic dismissal, and graduating, as well as by the student’s academic achievement at 
exit (GPA).This analysis answers the question, “Does the academy facilitate positive postsecondary 
educational outcomes for its students?” 

We empirically model a student’s entrance into the university as a function of career academy enroll- 
ment, individual demographics, high school of origin, and academic preparation, such that: 

5: entrance = a cnt + a® n, Acad. + a® n, GPA j + a“jDemo k + a'; nt £hs i 

where: 

entrance = a vector of variables measuring a student’s entrance into the university, 

hs = a vector of variables designating the student’s high school, and other variables are 
defined above. 

The coefficient on career academy (a. cnl ) is a measure of the direct association between the academy 
and the student’s probability of attending or enrolling in the university. The interpretation follows that 
outlined for the above equation. This estimation also allows us to trace the indirect influence of the 
career academy on postsecondary education through high-school GPA. If the career academy 
increases academic achievement in high school and high-school GPA increases the probability of en- 
trance into college (a ent > 0), the career academy indirecdy increases the probability of postsecondary 
education by increasing high-school GPA. 

We explicidy examine the direct and indirect (through GPA) influence of the academy in all 
outcomes in a summary of our results. For these analyses, we re-estimate equations without including 
GPA on the RHS (e.g., a em = 0). Within this formulation, the coefficient on career academy captures 
both the direct (e.g., a. cn ) and indirect impact through GPA (e.g., a g * a ent ).The indirect impact is 
then measured by the difference between the coefficient estimates without GPA (total effect) and 
with GPA (direct effect). 

Equation 5 is also used to examine the route that the student takes through the university by replacing 
the left-hand side of the equation with the need for remediation and transfer status. This modeling 
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implicitly argues that the factors used to admit and enroll students are correlated with the route 
through the university. Because acceptance to and enrollment in the university, remediation, and 
transfer status are all binary variables, we use probit analysis for all estimations of equation 5. 

Finally, we assess the academy’s influence on exit from college in terms of academic achievements in 
the university (GPA), dropping out, academic dismissal, and graduating. We initially model the stu- 
dents academic achievement at the university so that subsequent analysis can examine, through 
achievement in college, the direct and indirect influence of the academy. We model academic achieve- 
ment in college as a function of enrollment in a high-school career academy, academic achievement in 
high school, individual demographics, high school of origin, and the route through college. Both 
needing remediation and transferring to the university, the constructs for route through college, are 
negative influences because students often use these mechanisms to acquire academic skills that could 
have been learned in high school. Thus: 

6: collegegpa = a c + a'Acadj + a c GPA. + a^Demo k + a^hs, + a^£route m 

collegegpa = the student’s grade point average when leaving the university, 

route = a vector of variables designating a student’s route through the university (remediation 
and transfer), and other variables are defined above. 

The coefficient on career academy (a c ) estimates its independent, direct influence on academic 
achievement at the university and its interpretation follows that oudined above. The coefficient on the 
academic achievement in high school (<x[) allows us to trace the indirect influence of a career acad- 
emy on academic achievement in this university as outlined above. Since GPA is a relatively 
unbounded continuous measure, we estimate equation 3 with OLS. 

Our final model examines exit from college as a function of career academy enrollment, academic 
achievement in college, and individual demographics, such that: 

7: exit = a x + a^Acadj + a*collegegpa n + a x £ Demo k 

exit = a vector of variables indicating the type of exit from the university (dropping out, academic 
dismissal, and graduating), and other variables are defined above. 

The career academy’s influence on exit from college is modeled as both direct (0C. X ) and indirect, 
through college GPA (a* * 0 assuming <X C * 0). Our modeling suggests that neither high school nor 
route through college independendy influences exit from college. Rather, these factors indirecdy in- 
fluence a student’s exit from college through college GPA (equation 6). Because each of these exit 
categories is a binary measure, we estimate these equations with a probit analysis. 

Quantitative Data Sources 

Our primary source of information for the quantitative analysis is known as the City Student Data- 
base (CSDB), which was constructed for this study. This database followed a cohort of 10,110 students 
who were sophomores in the city’s public schools in 1990—1991, 1991— 1992, and 1992— 1993. About 
14% (1,407) of these students were enrolled in one of the districts nine career academies. The remain- 
ing 8,703 students, who were never enrolled in a career academy, became our “nonacademy” compar- 
ison group. The years of analysis covered in this study allowed for some time to elapse for high-school 
completion by spring 1996, when the post-high-school survey was sent. Quantitative information 
about these students was collected from two sources: district data files and post-high-school surveys. 
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District Data Files 



This database contains data from two different sources: student transcripts and general district files. The 
transcript data are a complete census of course work each student took in or transferred into the dis- 
trict. Information includes, for each course taken, the year and month of enrollment, course number 
and ride, course grade, and grade level of the student at the time that the course was taken. We used 
this data to construct individual GPAs, and a curricular history that included career academy course 
work compiled at the end of each semester (fall, winter, and summer). 

Because a students GPA is based on school records, we avoid biases that could result from individual 
reporting of grades. Data files were also constructed from each students demographic file. This in- 
cludes the student’s test score in each year in school, attendance in each semester, gender, race, and 
school. This information was linked to the transcript database with a unique identification number 
that was generated to prevent linkage with all other databases in the CSDB, a condition of the agree- 
ment to release the data for this study 

Post-High- School Surveys 

Data on students’ post-high-school activities were obtained from a survey that was mailed to each of 
the 10,110 students for whom we had district files. To enable this mailing, the district provided names, 
addresses, and the first and last school attended. Confidentiality prevented using district data files to 
identify career academy students in this file. Instead, we identified career academy students in this 
mailing from the names in the Local Evaluation Files. The post-high -school surveys were mailed to 
students at five high schools from May to October (with follow-ups lingering through December) of 
1996, when former students were between one and three years out of high school if they had gradu- 
ated on-time. 

Surveys were mailed to three “populations”: academy students at five schools (n=l,257), nonacademy 
students at five schools (n=6,804), and the students at the high school for which academy student 
status was not available in the Local Evaluation Files (n=2,041).The questionnaires that were sent to 
each population were virtually identical, with all of the differences limited to the first page and re- 
flecting specific differences in the respondent’s high-school program. For example, only career acad- 
emy students were asked about their internships and mentoring. A total of 1,228 surveys were 
returned: 981 from nonacademy students, and 247 from academy students. Response rates were higher 
for individuals in the academy sample. 

We intentionally focused questions on what was referred to in the survey instrument only as the stu- 
dent’s “high-school program.”The respondents’ self-determination of high-school programs allowed 
us to ask identical questions of academy and nonacademy students and to have them provide answers 
about high school that did not presuppose (from us) participation in any specific program structure or 
curriculum. 

If the questions had been differently worded for each group (e.g., “high-school programs” and 
“academy programs”), the responses might have reflected answers to different questions. For example, 
an academy student who wanted to focus high-school courses around college-preparatory activities 
may have been dissatisfied with the high-school program if the academy program was not sufficiendy 
scholarly. However, the student may have been satisfied with the academy experience overall because 
of its other qualities. 

The questionnaire allows for responses that cover both dimensions of the experience. The post-high- 
school questions were structured around five types of information: post-high-school educational out- 
comes; post-high-school labor-market information; expectations about education; the labor market 
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and the city as a place to live; assessment of the high school and program; and general demographic 
information. Respondents to the post-high-school survey had their responses linked to their transcript 
and district files to create the CSDB. 

Of critical import are the questions used to measure the dependent variables on lifelong learning. We 
used responses to the survey instruction: “Check the box that indicates how well your high-school 
education helped you to . . ” obtain 13 different work- and education-related types of knowledge and 
skills. Respondents were given four options with which to rate the efficacy of their high-school pro- 
gram: “a great deal,” “somewhat,” “a little,” or “not at all.”These measures were specifically designed to 
capture three sets of qualities: “workplace” skills, “education” skills, and “school-to-career” skills. We 
use the four “education” items to capture the career academy’s potential for building a capacity for 
lifelong learning. 

Data from the University 

For analyzing outcomes at the four-year university, we draw our data from a medium-size CSU using 
all applicants to degree programs who had graduated from a single district’s high schools between 
1990 and 1997. This data set includes information on all of the district’s graduates who 1) applied for 
admission, whether or not they completed the process; 2) were denied admission; 3) were admitted to 
the university but did not enroll; or 4) enrolled in the university. The data set contains information on 
individuals who had applied from 1990 through December 31, 1998, with data on university activities 
updated through spring 1999. In many ways, the university from which the data are drawn is typical 
of comprehensive, urban universities throughout the country. About 65% of the students reside within 
the county of the university’s main campus and another 19% reside in the county of its satellite cam- 
pus. No demographic group is a majority. 

University records include nearly complete information on student demographics, year and high 
school of graduation, and type of application (e.g., transfer and first-time freshman). High-school 
transcript information (GPA) is available for many applicants, although information is not always avail- 
able for individuals who did not complete the application process or who transferred from commu- 
nity colleges. For all students who enrolled in the university, information is available about the 
student’s exit status, their university GPA, and their major. We identified career academy students in 
this database with a listing of students from district records that was compiled as part of a mandated 
yearly evaluation of the program. 

Potential Estimation Caveats 

Our specification, modeling, and data use are compromises between alternative specifications that 
meet our evaluation goals and steps to minimize biases in parameter estimation. We examine potential 
issues and complications in estimations that our approach may present in order to more clearly define 
the limits of our analysis. 

Case Study 

The idiosyncrasies that exist within any case study can produce conclusions applicable only to one 
environment. While this critique cannot be dismissed out of hand for any case study, we believe that 
our methodology appropriately evaluates programs for educational reform. The district and city por- 
trayed in this study are typical of many inner cities and public schools throughout the country. They 
illustrate the obstacles facing reform efforts in any community by showing the political pressures and 
bureaucratic structures that exist. While the specifics of such pressures and structures are, indeed, idio- 
syncratic, their general form is in no way unique. 




46 



Sample Selection 

The primary criticism of our quantitative analysis would lie in the nonrandom assignment of students 
into the academy programs. This criticism arises because the “ideal” analysis of program outcomes is 
one in which “counterfactuals” are estimated. Outcomes from academy programs must be compared 
to those that would have existed for participants had the program not existed. If counterfactual out- 
comes are known, they could be compared to the actual outcomes that occurred under the career 
academies. This would assess the value of the academies. 

Of course, as with any study, counterfactuals are not known. Our study simply assumes that the out- 
comes for nonacademy participants with the same background characteristics provide an estimate of 
counterfactual outcomes. While this is far from perfect, we believe that it best suits the goal of this 
study: to assess the potential of the career academies as a “real world” educational reform. 

A typical method for quantifying counterfactual outcomes is the random assignment of individuals 
into control (e.g., nonacademy) and treatment (e.g., academy) programs. The control group provides 
an estimate of counterfactual outcomes. However, as we know, random assignment to high-school 
programs is not a policy option in the “real world” of high schools. Thus, program outcomes from 
such an experiment may or may not approximate those outcomes that will exist once the program is 
implemented. In an actual school district, students (or their parents) will be given a choice of pro- 
grams, and program selection becomes endogenous to program outcomes. The random assignment of 
students to control and treatment groups therefore will not reflect a potentially important dimension 
of choice that affects outcomes once the program is implemented. 

Still, we must address the issue of sample selection even if our goal is to reflect “real world” circum- 
stances. Without any controls for sample selection, our estimates of program outcomes would be dis- 
torted by differences in observed and unobserved characteristics of students in different programs. Our 
single-equation estimation of program outcomes without random assignment contains statistical con- 
trols for observed heterogeneity in students and implicit “controls” for unobserved heterogeneity. We 
checked for potential biases along observed heterogeneity lines by initially including controls for 
Heckman’s sample selection in 1) choosing an academy program; 2) taking initial period achievement 
tests (knowledge and skill control); and 3) responding to the survey. However, X was never significant 
and did not affect career academy coefficient estimates. 

Our strongest control for unobserved heterogeneity lies in our initial-period control for knowledge 
and skills. If the unobservables that increase educational outcomes in high school are capitalized into 
lOth-grade test scores, our parameter estimates of program impact will not contain biases from un- 
observables. Our inclusion of initial-period controls for knowledge and skills is akin to estimating a 
fixed-effect model. This is a common technique employed to purge estimates from biases that result 
from unobserved heterogeneity. We note that our analysis stratified by students’ initial academic 
knowledge also controls for unobserved heterogeneity by confining estimations to students with 
similar characteristics. 

Multicollinearity 

Our quantitative study relies on a value-added educational production function framework that mod- 
els educational outcomes in terms of educational inputs and processes. The career academy is the edu- 
cational process of interest. While differences existed among schools within the district in the degree 
to which the career academy model was implemented, the administrative adherence to the model 
caused curriculum components (e.g., integrated curriculum, internships) to be highly correlated. This 
correlation precludes including curriculum components directly into the parametric analysis. By con- 
fining our estimation to an aggregate measure of career academy (as outlined above), we cannot assess 
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the influence of the individual component with our quantitative analysis. We rely on the qualitative 
data for this assessment. 

Our modeling of the career academy educational process is therefore confined to the binary measures 
of enrollment in the individual career academies, which capture the totality of the subprocesses em- 
bedded within. By using binary variables for individual academies, we also keep open the possibility 
that the whole is greater than the sum of its parts. That is, the career academy educational process can 
be implemented in a variety of ways. For example, a program could have an emphasis on heavily in- 
tegrating its curriculum while placing less emphasis on work-based learning or, conversely, could place 
relatively more emphasis on work-based activities. 

We note that in the nonstratified analyses, in which a single academy binary variable measures pro- 
grams’ impact, our estimates of the influence of the career academy may be biased downward because 
of the inclusion of academies with incomplete programs. Since the coefficient on the binary academy 
variable includes the impact of weak programs, its size may increase when fully implemented program 
models are evaluated. This downward bias, however, may accurately measure the impact of the acad- 
emy as it is likely to occur in the “real world.” If political pressures and administrative structures in- 
fluence the development and implementation of career academy programs — and our study suggests 
that they do — then estimates of program outcomes of career academy students may reflect those that 
would occur in a situation where both complete and incomplete programs exist. 

Instrumental Variables 

Modeling the outcomes of education is complicated by the mediating role of family background. 

One common “solution” to this dilemma uses instrumental variables that are computed from esti- 
mations of family background to replace variables on R.HS (right-hand side) of the equations. This 
reduced-form approach models as instruments the factors that we model as inputs to educational 
outcomes. In this approach, the instruments reflect both the direct and indirect impacts of their pre- 
dictors. We use a structural approach to untangle these effects by using initial-period knowledge and 
skills to capture both academic abilities at high-school entrance and background factors. This im- 
plicitly assumes that the students’ academic skills at high-school entrance measure all background 
influences that would affect the knowledge and skills taken from high school. 

Error-terms Bias 

Unobserved variables that are correlated with independent or dependent variables can bias coefficient 
estimates. In research using educational production functions, this problem exists because many of 
these unmeasured variables vary systematically with school district (e.g., they reflect the environment 
surrounding the production). We minimize this type of bias because our single-school district repre- 
sents a relatively homogeneous educational environment. Because pupil-teacher ratios, per-pupil ex- 
penditures, class sizes, and most policies and procedures are all relatively constant, the estimated 
educational benefits associated with career academies are less likely to reflect spurious correlation than 
are estimates from a data set containing multiple districts. 
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NOTES 



1. Stern, 1994. 

2. State of California, 1995; OERI, 1994;USDOL, 1991. 

3. For example, see Ravitch, 1995 a, b. 

4. California Academy Support Network (http://casn.berkeley.edu). 

5. Stern, 1992, 1989, 1988; Dayton et aL, 1992. 

6. Dayton et al., 1992; Kemple and Snipes, 2000; Kemple, 1997; Kemple and Rock, 1996; Stern et aL, 1988. 

7. Kemple and Snipes, 2000. This research, conducted by the Manpower Demonstration Research Corporation, 
compared students who attended and did not attend selected academies in a number of different cities. In our other 
publication (Maxwell and Rubin, 2000) we discuss the advantages and limitations of the different methodological 
approaches. 

8. Maxwell and Rubin, 2000. 

9. Kemple and Snipes, 2000. 

10. Maxwell, forthcoming. 

11. HI1LE Center Report 99-3A is available by emailing hire@csuhayward.edu. 
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